{"id":230159,"date":"2023-06-20T11:33:57","date_gmt":"2023-06-20T08:33:57","guid":{"rendered":"https:\/\/isletgroup.fi\/?p=230159"},"modified":"2023-09-06T11:52:24","modified_gmt":"2023-09-06T08:52:24","slug":"planning-new-cloud-based-data-warehouse-solution","status":"publish","type":"post","link":"https:\/\/isletgroup.fi\/en\/2023\/06\/20\/planning-new-cloud-based-data-warehouse-solution\/","title":{"rendered":"Data ware\u00adhous\u00ading solu\u00adtion\u2019s jump to&nbsp;cloud"},"content":{"rendered":"<p>[et_\u200bpb_\u200bsection fb_built=\u201c1\u201d _builder_version=\u201c4.16\u201d _module_preset=\u201cdefault\u201d da_disable_devices=\u201coff|off|off\u201d global_\u200bcolors_\u200binfo=\u201d{}\u201d da_is_popup=\u201coff\u201d da_exit_intent=\u201coff\u201d da_has_close=\u201con\u201d da_alt_close=\u201coff\u201d da_dark_close=\u201coff\u201d da_not_modal=\u201con\u201d da_is_singular=\u201coff\u201d da_with_loader=\u201coff\u201d da_has_shadow=\u201con\u201d][et_pb_row _builder_version=\u201c4.16\u201d _module_preset=\u201cdefault\u201d global_colors_info=\u201d{}\u201d][et_pb_column type=\u201c4_4\u201d _builder_version=\u201c4.16\u201d _module_preset=\u201cdefault\u201d global_colors_info=\u201d{}\u201d][et_pb_text _builder_version=\u201c4.17.4\u201d _module_preset=\u201cdefault\u201d text_orientation=\u201cjustified\u201d global_colors_info=\u201d{}\u201d]<\/p>\n<p>Nowa\u00addays, data ware\u00adhous\u00ading solu\u00adtions large\u00adly rely on cloud resources. At Islet, a&nbsp;large por\u00adtion of new solu\u00adtions are imple\u00adment\u00aded as Lake\u00adhouse solu\u00adtions, either on top of Synapse Ana\u00adlyt\u00adics or Data\u00adbricks. The imple\u00admen\u00adta\u00adtion is deter\u00admined based on require\u00adments, and the data ware\u00adhous\u00ading solu\u00adtion is cho\u00adsen based on which tech\u00adnol\u00ado\u00adgy best ful\u00adfills these requirements.<\/p>\n<p>In rare cas\u00ades, the use of cloud ser\u00advices is ini\u00adti\u00adat\u00aded by bring\u00ading along part of the old solu\u00adtion. These few cas\u00ades are, for exam\u00adple, those where the orig\u00adi\u00adnal solu\u00adtion is so broad or impor\u00adtant to main\u00adtain con\u00adti\u00adnu\u00adity of oper\u00ada\u00adtions. We will go through the ways in which the tran\u00adsi\u00adtion to the cloud can be made by bring\u00ading along part of the old solution.<\/p>\n<p><\/p>\n<h3><strong>Ben\u00ade\u00adfits of&nbsp;cloud<\/strong><\/h3>\n<p>In this post, we will go through ways to start using Microsoft\u00ad\u2019s cloud ser\u00advices eas\u00adi\u00adly and effort\u00adless\u00adly in scope of data and ana\u00adlyt\u00adics. Cloud ser\u00advices offer sev\u00ader\u00adal ben\u00ade\u00adfits, such as flex\u00adi\u00adbil\u00adi\u00adty, per\u00adfor\u00admance, and cost-effectiveness.<\/p>\n<p>Flex\u00adi\u00adbil\u00adi\u00adty is one of the most sig\u00adnif\u00adi\u00adcant ben\u00ade\u00adfits of cloud ser\u00advices. Microsoft offers var\u00adi\u00adous ser\u00advices and solu\u00adtions that can be scaled accord\u00ading to your needs. You can eas\u00adi\u00adly increase or decrease capac\u00adi\u00adty and resources in line with your busi\u00adness require\u00adments. This gives you flex\u00adi\u00adbil\u00adi\u00adty and the abil\u00adi\u00adty to adapt quick\u00adly to rapid\u00adly chang\u00ading busi\u00adness conditions.<\/p>\n<p>Per\u00adfor\u00admance is anoth\u00ader impor\u00adtant advan\u00adtage that cloud ser\u00advices pro\u00advide. Microsoft\u00ad\u2019s cloud ser\u00advices are designed to pro\u00advide high per\u00adfor\u00admance and effi\u00adcient use of resources. You can lever\u00adage the scal\u00ada\u00adbil\u00adi\u00adty and per\u00adfor\u00admance offered by cloud ser\u00advices when deal\u00ading with large amounts of data or per\u00adform\u00ading heavy com\u00adpu\u00adta\u00adtion tasks. This ensures the effi\u00adcien\u00adcy and speed of your busi\u00adness operations.<\/p>\n<p>Cost-effec\u00adtive\u00adness is the third crit\u00adi\u00adcal fac\u00adtor that attracts busi\u00adness\u00ades to use cloud ser\u00advices. Microsoft\u00ad\u2019s cloud ser\u00advices offer flex\u00adi\u00adble pric\u00ading mod\u00adels where you pay only for the resources used. You don\u2019t need to invest a&nbsp;large ini\u00adtial cap\u00adi\u00adtal or main\u00adtain expen\u00adsive infra\u00adstruc\u00adtures. Along with scal\u00ada\u00adbil\u00adi\u00adty, cloud ser\u00advices also offer the oppor\u00adtu\u00adni\u00adty to opti\u00admize costs and pay only for the ser\u00advices you&nbsp;need.<\/p>\n<h3><\/h3>\n<h3><strong>Tra\u00addi\u00adtion\u00adal Data Ware\u00adhouse Solution<\/strong><\/h3>\n<p>A tra\u00addi\u00adtion\u00adal data ware\u00adhouse solu\u00adtion imple\u00adment\u00aded with Microsoft prod\u00aducts is usu\u00adal\u00adly based on the SQL Serv\u00ader, SSIS pack\u00adages for data trans\u00adfer, and sched\u00adul\u00ading with SQL Serv\u00ader&nbsp;Agent.<\/p>\n<p>SQL Serv\u00ader is a&nbsp;well-known and reli\u00adable data\u00adbase solu\u00adtion that pro\u00advides an effec\u00adtive foun\u00adda\u00adtion for a&nbsp;data ware\u00adhous\u00ading environment.<\/p>\n<p>SSIS (SQL Serv\u00ader Inte\u00adgra\u00adtion Ser\u00advices) is an ETL (Extract, Trans\u00adform, Load) tool pro\u00advid\u00aded by Microsoft, which sim\u00adpli\u00adfies data trans\u00adfer between dif\u00adfer\u00adent systems.<\/p>\n<p>Sched\u00adul\u00ading with the SQL Serv\u00ader Agent pro\u00advides automa\u00adtion and sched\u00adul\u00ading in the data ware\u00adhous\u00ading environment.<\/p>\n<p>This com\u00adbi\u00adna\u00adtion of SQL Serv\u00ader, SSIS pack\u00adages, and SQL Serv\u00ader Agent has pro\u00advid\u00aded a&nbsp;sol\u00adid foun\u00adda\u00adtion for data man\u00adage\u00adment and analysis.<\/p>\n<h3><\/h3>\n<h3><strong>How do we pro\u00adceed from this tra\u00addi\u00adtion\u00adal solu\u00adtion towards the&nbsp;cloud?<\/strong><\/h3>\n<p>The first step is to decide which parts of the cur\u00adrent data ware\u00adhouse you want to bring to the cloud. You can choose a&nbsp;cer\u00adtain or all parts depend\u00ading on your needs and busi\u00adness objec\u00adtives. It\u2019s impor\u00adtant to ensure and dou\u00adble check that the solu\u00adtions to be trans\u00adferred to the cloud are com\u00adpat\u00adi\u00adble with the new envi\u00adron\u00adment so that the tran\u00adsi\u00adtion is seamless.<\/p>\n<p>A tra\u00addi\u00adtion\u00adal data ware\u00adhouse solu\u00adtion imple\u00adment\u00aded with Microsoft prod\u00aducts is well-suit\u00aded for a&nbsp;lift-and-shift type of cloud migration.<\/p>\n<p>By mov\u00ading an SQL Serv\u00ader data ware\u00adhouse locat\u00aded on your own servers, as well as the SSIS runs, to the Microsoft\u00ad\u2019s Azure SQL Man\u00adaged Instance, the old solu\u00adtion can be migrat\u00aded with as lit\u00adtle effort as pos\u00adsi\u00adble, while also enabling the devel\u00adop\u00adment of new cloud-native solutions.<\/p>\n<p>Azure SQL Man\u00adaged Instance pro\u00advides near\u00adly 100% com\u00adpat\u00adi\u00adbil\u00adi\u00adty with local SQL Serv\u00ader, which facil\u00adi\u00adtates the tran\u00adsi\u00adtion from local data ware\u00adhous\u00ades to the cloud. In addi\u00adtion, exe\u00adcut\u00ading SSIS pack\u00adages in Azure Data Fac\u00adto\u00adry allows exist\u00ading ETL process\u00ades to be used with min\u00adi\u00admal changes.<\/p>\n<p>Cloud-based data ware\u00adhous\u00ades offer a&nbsp;more cost-effec\u00adtive way to scale resources as need\u00aded. Cus\u00adtomers can reduce costs by mov\u00ading from acquir\u00ading hard\u00adware used by the data ware\u00adhouse to only pay\u00ading for the resources used in the cloud. This avoids com\u00admit\u00adting sig\u00adnif\u00adi\u00adcant costs\u2026 for exam\u00adple every four years, and costs can be mon\u00adi\u00adtored and adjust\u00aded accord\u00ading to the resources used. Most resources in the Azure cloud offer scal\u00ada\u00adbil\u00adi\u00adty, as well as the option to shut down resources when they are not in&nbsp;use.<\/p>\n<p>Azure SQL Man\u00adaged Instance pro\u00advides scal\u00adable per\u00adfor\u00admance com\u00adpared to a&nbsp;local data ware\u00adhouse. Scal\u00adable resources ensure that sys\u00adtem capac\u00adi\u00adty adapts to cus\u00adtomer needs and enables effi\u00adcient work with large amounts of data. Rarely there are sit\u00adu\u00ada\u00adtions where the resources offered by Azure would no longer pro\u00advide scal\u00ada\u00adbil\u00adi\u00adty. The archi\u00adtec\u00adture must always con\u00adsid\u00ader the nature of dif\u00adfer\u00adent resources, the usage require\u00adments of the solu\u00adtion, and there\u00adby make the right resource choices.<\/p>\n<p>With cloud ser\u00advices, cus\u00adtomers can take advan\u00adtage of advanced resource secu\u00adri\u00adty that are con\u00adtin\u00adu\u00adous\u00adly updat\u00aded along with the ser\u00advice. In this case, the upkeep of keep\u00ading updates up to date is trans\u00adferred to the cloud ser\u00advice provider. Too often, local solu\u00adtions have lagged in updates or have even fall\u00aden out\u00adside of updates due to out\u00addat\u00aded oper\u00adat\u00ading systems.<\/p>\n<p><\/p>\n<h3><strong>Con\u00adtin\u00adu\u00adous Devel\u00adop\u00adment of Cloud-Native Solutions<\/strong><\/h3>\n<p>The solu\u00adtion does\u00adn\u2019t usu\u00adal\u00adly stop when moved to the cloud. This is only the first step towards cloud-native devel\u00adop\u00adment. To enable fur\u00adther devel\u00adop\u00adment, you can cre\u00adate new envi\u00adron\u00adments and use the ser\u00advices offered by the cloud even more efficiently.<\/p>\n<p>When cloud migra\u00adtion reach\u00ades the point where rapid tran\u00adsi\u00adtion to the cloud has been achieved based on the orig\u00adi\u00adnal solu\u00adtion, then we need to con\u00adsid\u00ader how the devel\u00adop\u00adment of the new solu\u00adtion will be imple\u00adment\u00aded. In this case, it makes sense to con\u00adtin\u00adue with a&nbsp;cloud-native solu\u00adtion, which means grad\u00adu\u00adal\u00adly mov\u00ading away from an SQL Serv\u00ader and SSIS-based solution.<\/p>\n<p>Below is an exam\u00adple of the ben\u00ade\u00adfits of resources used for cloud-native devel\u00adop\u00adment, for exam\u00adple if <strong>Azure Synapse<\/strong> is cho\u00adsen for cloud-native development.<\/p>\n<ol>\n<li><strong>Mas\u00adsive scal\u00ada\u00adbil\u00adi\u00adty:<\/strong> Data Lake-based solu\u00adtions are high\u00adly scal\u00adable, enabling the stor\u00adage and pro\u00adcess\u00ading of vast amounts of data. Azure Synapse, togeth\u00ader with Azure Data Lake Stor\u00adage Gen2, pro\u00advides high per\u00adfor\u00admance and scal\u00ada\u00adbil\u00adi\u00adty for both struc\u00adtured and unstruc\u00adtured&nbsp;data.<\/li>\n<li><strong>Flex\u00adi\u00adbil\u00adi\u00adty<\/strong>: Azure Synapse\u2019s Data Lake solu\u00adtion accepts var\u00adi\u00adous data types and for\u00admats, such as text files, images, videos, and JSON files. This flex\u00adi\u00adbil\u00adi\u00adty enables the han\u00addling and inte\u00adgra\u00adtion of com\u00adplex data struc\u00adtures, allow\u00ading com\u00adpa\u00adnies to ana\u00adlyze a&nbsp;wider range of data sources.<\/li>\n<li><strong>Cost-effi\u00adcien\u00adcy<\/strong>: Data Lake-based solu\u00adtions like Azure Synapse offer an afford\u00adable stor\u00adage option for large amounts of data. Stor\u00adage costs remain low because you only pay for the stor\u00adage space and com\u00adput\u00ading resources used.<\/li>\n<li><strong>Data man\u00adage\u00adment<\/strong>: Azure Synapse and Data Lake Stor\u00adage Gen2 togeth\u00ader offer an effec\u00adtive data man\u00adage\u00adment solu\u00adtion. A&nbsp;hier\u00adar\u00adchi\u00adcal stor\u00adage struc\u00adture and meta\u00adda\u00adta man\u00adage\u00adment enable eas\u00adi\u00ader clas\u00adsi\u00adfi\u00adca\u00adtion, retrieval, and orga\u00adni\u00adza\u00adtion of&nbsp;data.<\/li>\n<li><strong>Data secu\u00adri\u00adty and com\u00adpli\u00adance<\/strong>: Azure Synapse includes advanced secu\u00adri\u00adty fea\u00adtures such as data encryp\u00adtion at rest and in tran\u00adsit. In addi\u00adtion, it adheres to strict data pro\u00adtec\u00adtion stan\u00addards, such as GDPR, help\u00ading com\u00adpa\u00adnies ensure data pro\u00adtec\u00adtion and compliance.<\/li>\n<li><strong>Real-time ana\u00adlyt\u00adics<\/strong>: Azure Synapse and Data Lake solu\u00adtion enable real-time ana\u00adlyt\u00adics in han\u00addling large amounts of data. This allows for quick insights and sup\u00adports data-dri\u00adven deci\u00adsion making.<\/li>\n<li><strong>Com\u00adpat\u00adi\u00adbil\u00adi\u00adty and inte\u00adgra\u00adtion:<\/strong> Azure Synapse is com\u00adpat\u00adi\u00adble with many cloud ser\u00advices and tools, includ\u00ading Azure Machine Learn\u00ading, Pow\u00ader BI, and Azure Data Fac\u00adto\u00adry. This facil\u00adi\u00adtates data flow man\u00adage\u00adment, ana\u00adlyt\u00adics, and visualization.<\/li>\n<\/ol>\n<p><\/p>\n<p>To enable cloud-native devel\u00adop\u00adment, it is cru\u00adcial to cre\u00adate new envi\u00adron\u00adments and uti\u00adlize cloud ser\u00advices such as Azure Synapse and Data Lake. With these, you can con\u00adtin\u00adue to devel\u00adop your solu\u00adtion, scale resources as need\u00aded, and lever\u00adage the full ben\u00ade\u00adfits offered by the&nbsp;cloud.<\/p>\n<p>Microsoft also offers oth\u00ader cloud-native devel\u00adop\u00adment mod\u00adels to sup\u00adport data and ana\u00adlyt\u00adics solu\u00adtions. You might see some\u00adthing about those in our Islet blogs\u2026 stay&nbsp;tuned!<\/p>\n<p><\/p>\n<p>Best regards,<\/p>\n<p>Mikko and&nbsp;Ilkka<\/p>\n<p><\/p>\n<p>PS. If inter\u00adest\u00aded, read more about our Data &amp;&nbsp;Ana\u00adlyt\u00adics ser\u00advices <strong><a href=\"https:\/\/isletgroup.fi\/en\/data-analytics-services\/\">here<\/a><\/strong><\/p>\n<div class=\"et_pb_row et_pb_row_0\">\n<div class=\"et_pb_column et_pb_column_4_4 et_pb_column_0  et_pb_css_mix_blend_mode_passthrough et-last-child\">\n<div class=\"et_pb_module et_pb_text et_pb_text_1  et_pb_text_align_left et_pb_bg_layout_light\">\n<div class=\"et_pb_text_inner\">\n<p><\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"et_pb_row et_pb_row_1\">\n<div class=\"et_pb_column et_pb_column_4_4 et_pb_column_1  et_pb_css_mix_blend_mode_passthrough et-last-child\">\n<div class=\"et_pb_module et_pb_text et_pb_text_2  et_pb_text_align_left et_pb_bg_layout_light\">\n<div class=\"et_pb_text_inner\">\n<p>#Islet\u00adGroup #data #ana\u00adlyt\u00adics #Pow\u00ader\u00adPlat\u00adform #Dat\u00ada\u00adPlat\u00adform #Microsoft #ware\u00adhous\u00ading #cloud<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<ol>\n<li style=\"list-style-type: none;\"><\/li>\n<\/ol>\n<p>[\/et_pb_text][\/et_pb_column][\/et_pb_row][\/et_pb_section]<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Nowa\u00addays, data ware\u00adhous\u00ading solu\u00adtions large\u00adly rely on cloud resources. At Islet, a&nbsp;large por\u00adtion of new solu\u00adtions are imple\u00adment\u00aded as Lake\u00adhouse solu\u00adtions, either on top of Synapse Ana\u00adlyt\u00adics or Data\u00adbricks. The imple\u00admen\u00adta\u00adtion is deter\u00admined based on require\u00adments, and the data ware\u00adhous\u00ading solu\u00adtion is cho\u00adsen based on which tech\u00adnol\u00ado\u00adgy best ful\u00adfills these require\u00adments. In rare&nbsp;cases,&nbsp;[\u2026]<\/p>\n","protected":false},"author":20,"featured_media":230504,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_et_pb_use_builder":"on","_et_pb_old_content":"<p>[et_pb_section fb_built=\"1\" _builder_version=\"4.16\" _module_preset=\"default\" da_disable_devices=\"off|off|off\" global_colors_info=\"{}\" theme_builder_area=\"post_content\"][et_pb_row _builder_version=\"4.16\" _module_preset=\"default\" global_colors_info=\"{}\" theme_builder_area=\"post_content\"][et_pb_column type=\"4_4\" _builder_version=\"4.16\" _module_preset=\"default\" global_colors_info=\"{}\" theme_builder_area=\"post_content\"][et_pb_text _builder_version=\"4.17.4\" _module_preset=\"default\" text_orientation=\"justified\" hover_enabled=\"0\" global_colors_info=\"{}\" theme_builder_area=\"post_content\" sticky_enabled=\"0\"]<\/p><p><strong>T\u00e4ss\u00e4 postauksessa k\u00e4ymme l\u00e4pi tapoja aloittaa Microsoftin pilvipalveluiden k\u00e4ytt\u00f6 helposti ja vaivattomasti datan ja analytiikan n\u00e4k\u00f6kulmasta. Pilvipalvelut tarjoavat useita etuja, kuten joustavuutta, suorituskyky\u00e4 ja kustannustehokkuutta.<\/strong><\/p><p>Joustavuus on yksi merkitt\u00e4vimmist\u00e4 pilvipalveluiden eduista. Microsoft tarjoaa useita erilaisia palveluita ja ratkaisuja, jotka voidaan skaalata tarpeen mukaan. Voit helposti lis\u00e4t\u00e4 tai v\u00e4hent\u00e4\u00e4 kapasiteettia ja resursseja yrityksesi tarpeiden mukaisesti. T\u00e4m\u00e4 antaa sinulle joustavuutta ja mahdollisuuden sopeutua nopeasti muuttuviin liiketoimintaolosuhteisiin.<\/p><p>Suorituskyky on toinen t\u00e4rke\u00e4 etu, jonka pilvipalvelut tarjoavat. Microsoftin pilvipalvelut ovat suunniteltu tarjoamaan korkeaa suorituskyky\u00e4 ja tehokasta resurssien k\u00e4ytt\u00f6\u00e4. Voit hy\u00f6dynt\u00e4\u00e4 pilvipalveluiden tarjoamaa skaalautuvuutta ja suorituskyky\u00e4 esimerkiksi k\u00e4sitelless\u00e4si suuria datam\u00e4\u00e4ri\u00e4 tai suorittaessasi raskaita laskentateht\u00e4vi\u00e4. N\u00e4in voit varmistaa yrityksesi toiminnan tehokkuuden ja nopeuden.<\/p><p>Kustannustehokkuus on kolmas t\u00e4rke\u00e4 tekij\u00e4, joka houkuttelee yrityksi\u00e4 siirtym\u00e4\u00e4n pilvipalveluiden k\u00e4ytt\u00f6\u00f6n. Microsoftin pilvipalvelut tarjoavat joustavat hinnoittelumallit, joissa maksat vain k\u00e4ytetyist\u00e4 resursseista. Sinun ei tarvitse investoida suurta alkup\u00e4\u00e4omaan tai yll\u00e4pit\u00e4\u00e4 kalliita infrastruktuureja. Pilvipalvelut tarjoavat skaalautuvuuden my\u00f6t\u00e4 my\u00f6s mahdollisuuden optimoida kustannuksia ja maksaa ainoastaan tarvitsemistasi palveluista.<\/p><h3>\u00a0<\/h3><h3><strong>Perinteinen tietovarastoratkaisu<\/strong><\/h3><p>Perinteinen Microsoftin tuotteilla toteutettu tietovarastoratkaisu perustuu yleens\u00e4 SQL Server -palvelimeen, SSIS-paketteihin tiedonsiirtoon sek\u00e4 ajastuksiin SQL Server Agentin kanssa.<\/p><p>SQL Server on tunnettu ja luotettava tietokantaratkaisu, joka tarjoaa tehokkaan perustan tietovarastointiymp\u00e4rist\u00f6lle.<\/p><p>SSIS (SQL Server Integration Services) on Microsoftin tarjoama ETL (Extract, Transform, Load) -ty\u00f6kalu, joka helpottaa tiedonsiirtoa eri j\u00e4rjestelmien v\u00e4lill\u00e4.<\/p><p>Ajastukset SQL Server Agentin kanssa tarjoavat automaatiota ja aikataulutusta tietovarastointiymp\u00e4rist\u00f6ss\u00e4.<\/p><p>T\u00e4m\u00e4 yhdistelm\u00e4 SQL Serveri\u00e4, SSIS-paketteja ja SQL Server Agenttia on tarjonnut vankan perustan tiedonhallinnalle ja analysoinnille.<\/p><h3>\u00a0<\/h3><h3><strong>Ty\u00f6nkulun luonti ja riippuvuuksien hallinta<\/strong><\/h3><p>Viimeaikaisena uutuutena Databricksiin tullut <a href=\"https:\/\/www.databricks.com\/product\/delta-live-tables\"><strong>Delta Live Table<\/strong><\/a> (DLT) tarjoaa my\u00f6s huomattavaa lis\u00e4arvoa dataratkaisuille. Mielest\u00e4ni olennaisimpana lis\u00e4n\u00e4 t\u00e4m\u00e4 tuo automaattisen DAG-tyypisen ty\u00f6nkulun muodostuksen sek\u00e4 riippuvuuksien hallinnan. T\u00e4m\u00e4 auttaa pit\u00e4m\u00e4\u00e4n prosessointiketjut oikeassa j\u00e4rjestyksess\u00e4 ja varmistaa, ett\u00e4 oikeat taulut latautuvat oikeassa j\u00e4rjestyksess\u00e4. DAG-ketjut ovat my\u00f6s k\u00e4ytt\u00e4jien n\u00e4ht\u00e4vill\u00e4, jolloin datan muutoshistoria (eng. Data lineage) on n\u00e4ht\u00e4vill\u00e4 raakatauluista aina raportointiin asti. Lis\u00e4ksi DLT mahdollistaa deklaratiivisen dataputkikehityksen miss\u00e4 sama dataputki ja sama koodi pystyy prosessoimaan eri l\u00e4hdetauluja joko er\u00e4ajona tai striimin\u00e4, jolloin arkkitehtuuri yksinkertaistuu ja koodin m\u00e4\u00e4r\u00e4 v\u00e4henee. DLT-dataputkien kautta kulkevan datan laadunvalvonta on samalla tavoin pitk\u00e4lle automatisoitu, miss\u00e4 ennalta m\u00e4\u00e4riteltyjen s\u00e4\u00e4nt\u00f6jen ja ehtojen perusteella data validoidaan prosessoinnin yhteydess\u00e4 ja tulokset, kuten ehtoja ei-t\u00e4ytt\u00e4vien rivien m\u00e4\u00e4r\u00e4, on visuaalisesti n\u00e4kyvill\u00e4 k\u00e4ytt\u00e4j\u00e4lle.<\/p><p>\u00a0<\/p><h3><strong>Miten t\u00e4st\u00e4 perinteisest\u00e4 ratkaisusta l\u00e4hdet\u00e4\u00e4n eteenp\u00e4in kohti pilve\u00e4?<\/strong><\/h3><p>Ensimm\u00e4inen askel on p\u00e4\u00e4tt\u00e4\u00e4, mit\u00e4 osia nykyisest\u00e4 tietovarastosta halutaan siirt\u00e4\u00e4 pilveen. Voit valita tietyn osan tai kaikki osat riippuen tarpeistasi ja liiketoiminnallisista tavoitteistasi. T\u00e4rke\u00e4\u00e4 on varmistaa, ett\u00e4 pilveen siirrett\u00e4v\u00e4t ratkaisut ovat yhteensopivia uuden ymp\u00e4rist\u00f6n kanssa, jotta siirtyminen tapahtuu saumattomasti.<\/p><p>Microsoftin tuotteilla toteutettu perinteinen tietovarastoratkaisu soveltuu hyvin nosto ja siirto (lift-and-shift)- tyyppiseen pilveist\u00e4miseen.<\/p><p>Siirt\u00e4m\u00e4ll\u00e4 omilla palvelimilla sijaitsevan SQL Server- tietovaraston, sek\u00e4 SSIS- ajot Microsoftin Azure SQL Managed- instanssille, sek\u00e4 ottamalla k\u00e4ytt\u00f6\u00f6n Azure Data Factoryn sek\u00e4 my\u00f6s Azure Synapsen, saadaan vanha ratkaisu ajautumaan mahdollisimman v\u00e4h\u00e4ll\u00e4 vaivalla, sek\u00e4 mahdollistetaan uuden pilvinatiivin kehitt\u00e4misen tekeminen.<\/p><p>Azure SQL Managed Instance tarjoaa l\u00e4hes 100% yhteensopivuuden paikallisen SQL Serverin kanssa, mik\u00e4 helpottaa siirtymist\u00e4 paikallisista tietovarastoista pilveen. Lis\u00e4ksi SSIS-pakettien suorittaminen Azure Data Factoryss\u00e4 mahdollistaa olemassa olevien ETL-prosessien k\u00e4ytt\u00e4misen minimaalisin muutoksin.<\/p><p>Pilvipohjainen tietovarasto tarjoaa kustannustehokkaamman tavan skaalata resursseja tarpeen mukaan. Asiakkaat voivat v\u00e4hent\u00e4\u00e4 kustannuksia siirt\u00e4m\u00e4ll\u00e4 tietovaraston k\u00e4ytt\u00e4m\u00e4n raudan hankinnasta maksamalla vain k\u00e4ytetyist\u00e4 resursseista pilvess\u00e4. T\u00e4ll\u00f6in ei tarvitse sitoutua kerran nelj\u00e4ss\u00e4 vuodessa mittaviin kustannuksiin, vaan kustannuksia voidaan seurata ja s\u00e4\u00e4t\u00e4\u00e4 k\u00e4ytettyjen resurssien mukaisesti. Useimmat resurssit Azuren- pilvess\u00e4 tarjoavat skaalautuvuutta, sek\u00e4 mahdollisuuden sammuttaa resurssit silloin kun niit\u00e4 ei k\u00e4ytet\u00e4<\/p><p>Azure SQL Managed Instance tarjoaa skaalautuvan suorituskyvyn paikalliseen tietovarastoon verrattuna. Skaalautuvat resurssit takaavat, ett\u00e4 j\u00e4rjestelm\u00e4 kapasiteetti mukautuu asiakkaiden tarpeisiin ja mahdollistaa tehokkaan ty\u00f6skentelyn suurten tietom\u00e4\u00e4rien kanssa. Harvoin tulee tilanteita, milloin Azuren resursseja tarjoamat resurssit eiv\u00e4t tarjoaisi en\u00e4\u00e4 lis\u00e4\u00e4 skaalautuvuutta. Arkkitehtuurissa t\u00e4ytyy aina ottaa huomioon eri resurssien luonne, ratkaisun k\u00e4ytt\u00f6vaatimukset ja sit\u00e4 kautta tehd\u00e4 oikeanlaiset resurssivalinnat.<\/p><p>Pilvipalveluiden avulla asiakkaat voivat hy\u00f6dynt\u00e4\u00e4 kehittyneit\u00e4 tietoturvaratkaisuja, jotka p\u00e4ivittyv\u00e4t palvelun mukana. T\u00e4ll\u00f6in huolenpito p\u00e4ivityksien ajan tasalla olosta siirtyy pilvipalvelun yll\u00e4pidett\u00e4v\u00e4ksi. Liian usein paikalliset ratkaisut ovat j\u00e4\u00e4neet p\u00e4ivityksiss\u00e4 j\u00e4lkeen tai ovat jopa tippuneet p\u00e4ivitysten ulkopuolelle k\u00e4ytt\u00f6j\u00e4rjestelmien vanhentumisten takia.<\/p><p>\u00a0<\/p><h3><strong>Pilvinatiivin ratkaisun jatkokehitys<\/strong><\/h3><p>Pilveistetty ratkaisu ei suinkaan pys\u00e4hdy siirt\u00e4miseen pilveen. Se on vasta ensimm\u00e4inen askel matkalla kohti pilvinatiivia kehityst\u00e4. Jatkokehityksen mahdollistamiseksi voit luoda uusia ymp\u00e4rist\u00f6j\u00e4 ja hy\u00f6dynt\u00e4\u00e4 pilven tarjoamia palveluita entist\u00e4 tehokkaammin.<\/p><p>Kun pilveist\u00e4misess\u00e4 p\u00e4\u00e4st\u00e4\u00e4n siihen, ett\u00e4 nopea siirtyminen pilveen on saavutettu alkuper\u00e4isen ratkaisun pohjalta, niin t\u00e4m\u00e4n j\u00e4lkeen joudutaan miettim\u00e4\u00e4n, ett\u00e4 miten uuden kehitt\u00e4minen toteutetaan. T\u00e4ll\u00f6in ratkaisu on j\u00e4rkev\u00e4\u00e4 jatkaa tekem\u00e4ll\u00e4 pilvinatiivia ratkaisua, joka tarkoittaa siirtymist\u00e4 SQL Server ja SSIS- pohjaisesta ratkaisusta v\u00e4hitellen poisp\u00e4in.<\/p><p>Ohessa esimerkki pilvinatiiviin kehitt\u00e4miseen k\u00e4ytettyjen resurssien hy\u00f6dyist\u00e4, jos valitaan pilvinatiiviin kehitt\u00e4miseen esim. Azure Synapse.<\/p><p>Azure Synapsen Data Lake -pohjaisen ratkaisun edut:<\/p><ol><li>Massiivinen skaalautuvuus: Data Lake -pohjaiset ratkaisut ovat eritt\u00e4in skaalautuvia, mik\u00e4 mahdollistaa valtavien tietom\u00e4\u00e4rien tallentamisen ja k\u00e4sittelyn. Azure Synapse yhdess\u00e4 Azure Data Lake Storage Gen2:n kanssa tarjoaa korkean suorituskyvyn ja skaalautuvuuden sek\u00e4 rakenteellisille ett\u00e4 rakenteettomille tiedoille.<\/li><li>Joustavuus: Azure Synapsen Data Lake -ratkaisu hyv\u00e4ksyy erilaisia tietotyyppej\u00e4 ja -formaatteja, kuten tekstitiedostoja, kuvia, videoita ja JSON-tiedostoja. T\u00e4m\u00e4 joustavuus mahdollistaa monimutkaisten tietorakenteiden k\u00e4sittelyn ja integraation, mink\u00e4 ansiosta yritykset voivat analysoida monipuolisempia tietol\u00e4hteit\u00e4.<\/li><li>Kustannustehokkuus: Data Lake -pohjaiset ratkaisut, kuten Azure Synapse, tarjoavat edullisen tallennusvaihtoehdon suurille tietom\u00e4\u00e4rille. Tallennuskustannukset pysyv\u00e4t alhaisina, koska voit maksaa vain k\u00e4ytetyst\u00e4 tallennustilasta ja laskentaresursseista.<\/li><li>Tietojen hallinta: Azure Synapse ja Data Lake Storage Gen2 yhdess\u00e4 tarjoavat tehokkaan tietojenhallintaratkaisun. Hierarkkinen tallennusrakenne ja metadata-hallinta mahdollistavat tietojen luokittelun, haun ja j\u00e4rjest\u00e4misen entist\u00e4 helpommin.<\/li><li>Tietojen suojaus ja vaatimustenmukaisuus: Azure Synapse -ratkaisu sis\u00e4lt\u00e4\u00e4 kehittyneit\u00e4 tietoturvaominaisuuksia, kuten tietojen salauksen lepotilassa ja liikkuessa. Lis\u00e4ksi se noudattaa tiukkoja tietosuojastandardeja, kuten GDPR:\u00e4\u00e4, auttaen yrityksi\u00e4 varmistamaan tietojen suojaamisen ja vaatimustenmukaisuuden.<\/li><li>Reaaliaikainen analytiikka: Azure Synapse ja Data Lake -ratkaisu mahdollistavat reaaliaikaisen analytiikan suurten tietom\u00e4\u00e4rien k\u00e4sittelyss\u00e4. T\u00e4m\u00e4 mahdollistaa nopeat oivallukset ja tietoon perustuvan p\u00e4\u00e4t\u00f6ksenteon tukemisen.<\/li><li>Yhteensopivuus ja integraatio: Azure Synapse on yhteensopiva monien pilvipalveluiden ja ty\u00f6kalujen kanssa, mukaan lukien Azure Machine Learning, Power BI ja Azure Data Factory. T\u00e4m\u00e4 helpottaa tietovirtojen hallintaa, analytiikkaa ja visualisointia.<\/li><\/ol><p>Pilvinatiivin kehityksen mahdollistamiseksi on siis t\u00e4rke\u00e4\u00e4 luoda uusia ymp\u00e4rist\u00f6j\u00e4 ja hy\u00f6dynt\u00e4\u00e4 pilvipalveluita kuten Azure Synapse ja Datalake. N\u00e4iden avulla voit jatkaa ratkaisusi kehitt\u00e4mist\u00e4, skaalata resursseja tarpeen mukaan ja hy\u00f6dynt\u00e4\u00e4 pilven tarjoamia etuja.<\/p><p>Microsoft tarjoaa my\u00f6s muita pilvinatiiveja kehitysmalleja data ja analytiikkaratkaisuiden tueksi, joita saatamme k\u00e4yd\u00e4 l\u00e4pi my\u00f6s muissa Isletin blogeissamme. Kannattaa siis seurata tarkkaan!<\/p><ol><li style=\"list-style-type: none;\">\u00a0<\/li><\/ol><p>[\/et_pb_text][\/et_pb_column][\/et_pb_row][et_pb_row _builder_version=\"4.16\" _module_preset=\"default\" global_colors_info=\"{}\" theme_builder_area=\"post_content\"][et_pb_column type=\"4_4\" _builder_version=\"4.16\" _module_preset=\"default\" global_colors_info=\"{}\" theme_builder_area=\"post_content\"][et_pb_text _builder_version=\"4.17.4\" _module_preset=\"default\" hover_enabled=\"0\" global_colors_info=\"{}\" theme_builder_area=\"post_content\" sticky_enabled=\"0\"]<\/p><p>Parhain terveisin,<\/p><p>Ilkka ja Mikko<\/p><p>\u00a0<\/p><p>PS. Jos kiinnostuit, <strong><a href=\"https:\/\/isletgroup.fi\/data-ja-analytiikkapalvelut\/\">t\u00e4\u00e4lt\u00e4 voit lukea lis\u00e4\u00e4 data- ja analytiikkapuolen palveluistamme.<\/a><\/strong><\/p><p>\u00a0<\/p><div class=\"et_pb_row et_pb_row_1\"><div class=\"et_pb_column et_pb_column_4_4 et_pb_column_1  et_pb_css_mix_blend_mode_passthrough et-last-child\"><div class=\"et_pb_module et_pb_text et_pb_text_2  et_pb_text_align_left et_pb_bg_layout_light\"><div class=\"et_pb_text_inner\"><p>#IsletGroup #data #analytiikka #PowerPlatform #DataPlatform #Microsoft #tiedollajohtaminen #raportointi #tietovarastointi<\/p><\/div><\/div><\/div><\/div><p>[\/et_pb_text][\/et_pb_column][\/et_pb_row][\/et_pb_section]<\/p>","_et_gb_content_width":"","wp_typography_post_enhancements_disabled":false,"footnotes":""},"categories":[453],"tags":[],"class_list":["post-230159","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-cover-story"],"acf":[],"_links":{"self":[{"href":"https:\/\/isletgroup.fi\/en\/wp-json\/wp\/v2\/posts\/230159","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/isletgroup.fi\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/isletgroup.fi\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/isletgroup.fi\/en\/wp-json\/wp\/v2\/users\/20"}],"replies":[{"embeddable":true,"href":"https:\/\/isletgroup.fi\/en\/wp-json\/wp\/v2\/comments?post=230159"}],"version-history":[{"count":11,"href":"https:\/\/isletgroup.fi\/en\/wp-json\/wp\/v2\/posts\/230159\/revisions"}],"predecessor-version":[{"id":230256,"href":"https:\/\/isletgroup.fi\/en\/wp-json\/wp\/v2\/posts\/230159\/revisions\/230256"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/isletgroup.fi\/en\/wp-json\/wp\/v2\/media\/230504"}],"wp:attachment":[{"href":"https:\/\/isletgroup.fi\/en\/wp-json\/wp\/v2\/media?parent=230159"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/isletgroup.fi\/en\/wp-json\/wp\/v2\/categories?post=230159"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/isletgroup.fi\/en\/wp-json\/wp\/v2\/tags?post=230159"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}