Let’s dive into the world of Microsoft Data Platform and Power Platform, and specifically the events and most impactful releases in our view, Islet’s Data & Analytics team members Ilkka Ellilä and Mikko Anttila. How do we leverage Microsoft cloud services to support our customers’ business and success? What are the benefits and how can they be utilized in different implementations? What are the most anticipated new features in these services in the near future and why?
We both have vast experience in using the Microsoft ecosystem in data warehousing projects, coding experience from the past and we both have been Microsoft certified in areas of our personal interests. Mikko has been specializing into the Data Platform side, and Ilkka gaining knowledge of the fast-growing part of low-code development in Power Platform and how it can be harnessed to boost effectiveness in the Data & Analytics environment and streamline business users, data engineers or analysts’ workflows.
Year 2022 was full of new releases in Microsoft Data and Power Platform areas. Thoughts were thrown around in Teams about new features and we came close to testing them all… Here are our most important findings:
POWER APPS
Power Apps had some great releases in 2022 regarding especially the development of PowerApps Canvas app, but first I think I should cast some light on Power Platform development within data warehousing environment in general.
- Power Platform provides all the tricks in the book for data automation, integration via connectors and even certain artificial intelligence tasks.
- Data Warehousing lacks often a good way of implementing customer driven data input. In most cases “the only way” has been SharePoint or Excel which are great for what they’re intended for, but handling long lists of user input data will cause problems at some point of a data warehousing project.
When Power Platform is needed in projects, we use Power Apps Canvas with SQL connector and Power Automate (Flow) in most cases.
Best or most discussed things (2022) – Power Apps
- Search and Find and Replace
Search and Find and Replace in general might sound not that new and ground breaking, but in fact, the PowerApps Canvas app was lacking just that. It changed renaming, code checking, variable handling, theming etc. from agony to easy-mode. It has really made the whole development experience way better.
- AI in general
AI was the big topic at the end of year 2022. AI seems to be the answer for everything.
Even PowerApps Canvas had its share of AI. Now you can code even less with Power Apps Ideas. Ideas enables the developer to just write the wanted ability that PowerFx should do and Ideas provides the commented code for it. I have not personally tested this ability broadly because my most used PowerFx commands are already quite clear – although if there were a lot to write and the use-case was presented in clear manner for Ideas, it could lead to even less time spent coding… These are still kind of mind-boggling abilities but clearly the future of coding.
Also, if you decide to code yourself, then it can help fixing the problems with Formula Repair (preview)
- Small changes in environment handling
Environment grouping and changes in keeping the environment active is pretty handy when working with many different environments and it really helps with the workflow. There was nothing more frustrating than to deselect the Default environment every…, single…, time, when logging into make.powerapps.com.
Things to look forward to (2023) – Power Apps
- Making teamwork easier with Co-authoring, Git and commenting
Usually, Coding project in general does include multiple developers doing simultaneous development work. Co-authoring feature and Git repository use can help with this. Git integration helps with Canvas app and Modern app.
With Canvas app you will be able to use Git and therefore make use of Git’s powerful features while development. Model driven apps will have an Office-like “Coauthoring” experience where you see every change made by your co-workers in real-time.
Comments are great way to add notes and questions about the code and share information inside a team developers.
- Named formulas
Everyone doing Canvas development knows App.OnStart and what it can do and what certain problems it can have regarding to the start-times of the app. Usually the problems appear when having multiple global parameters set-up on application start. If you are fetching data from the database layer, and setting that data to parameters, then the data download times may cause delays is actual application start.
Named Formulas will streamline the use of basic parameters by setting those as named formulas. When using the formula in your application it will have the values ready to go but loading those won’t affect the actual application start.
- App from image – UI creation with great possibilities
You can create a working app UI from your team’s planning whiteboard by just taking a picture and uploading it. This feature is at a “fun demo” stage at the moment, but it can really become something in the future. Tried it and yes, it created the app, but some texts were not read correctly, and some boxes were missed. Did my best to have as basic boxes and font as I could. For demo purposes I will most certainly keep this in my playbook.
And if you really want to head start planning your UIs with something else than Canvas app Designer, with Figma files you can create the app’s every screen. UI specialists can still use their Figma and you can just ask them to create a new cool UI.
- Managed Environments pipelines
Automated deployment pipelines have been used in development projects to automate the solutions deployments into the new environments. Most important aspect has been improved visibility about the situation between environments and the history of deployments.
PowerPlatform pipelines will be its own installable application that you install into the selected environment. With the app you can manage all the environments and see the audit logs, deployment activities, access errors and much more.
POWER BI
Power BI development is and will be an important part of Data & Analytics teams’ work. Models, measures, and visualizations do not happen by themselves, although AI can nowadays make nice suggestions, but still good analytical insights into the data context are needed to make reporting solutions work correctly.
There aren’t many as fast evolving reporting environments as Power BI around, so keeping up with new features is important.
Best or most discussed things (2022) – Power BI
- Composite models upgraded
Creating PowerBI datasets has not always been as flexible as now, and more improvements are coming. The composite models have proven to be good way to implement datasets combining once-a-day-refreshing, “data warehousing” data, and daily changing user input (PowerApps etc). Look and feel are that of a real-time report with this “wow” factor. We’ve used this for many datasets and it always strikes me as “look, how cool is this?”
The feature has more releases coming and now we are able to use the Azure Analysis Services models in composite model.
- Dataset refresh cancellation now available in Power BI Premium
Refreshing a big dataset can be time-consuming. In Power BI Pro workspace the maximum dataset size is 1GB and usually that maximum capacity is not met, or you would be really stretching the capacity.
In Premium capacity the datasets can be a lot bigger, therefore if you had miss-clicked a dataset refresh, the processing can take ages. It is very convenient to be able to cancel the current dataset refresh.
Things to look forward to (2023) – Power BI
- Azure Analysis Services automated migration to Power BI
Power BI Premium has been adding features into its datasets and is now pretty close to Azure Analysis Services. Microsoft announced an automated migration tool that allows users to migrate their Azure Analysis Services models into Power BI datasets. Later they will add a Power BI report rebind tool, but for now the report rebinding must be done manually.
With models in Power BI Premium you have a centralized place for the data and enjoy cost-related benefits.
- Cross-tenant Power BI Dataset Sharing
Soon you can create datasets in the Power BI service and then share those to users in a different tenant. It has a lot of real-world use cases for example in business-to-business data sharing, and data collaboration also offers business cases for new features.
AZURE DATA PLATFORM
Azure Data Platform is the backbone of a data warehousing solution. Continuous feature upgrades and new ways to implement working solutions will keep development work compelling and meaningful.
Best or most discussed things (2022) – Azure Data Platform
- Azure Machine Learning – Automated ML
Automated ML provides a user less experienced with forecasting models, some good tools to offer usable, code-commented, models. It won’t remove the need for data scientists because the model is just a start that requires tweaks and making sure that the forecasting models are optimized for more accurate or complex use cases.
After testing the feature, I was amazed as there were a large number of models and each model could be given a commented version if desired. On the whole, everything was well documented, only some undocumented features remained.
- Azure Data Factory UI changes
One of the most noticeable new things in Azure Data Factory was that the validation for certain components were made more intelligent. Once you could use some components nested and continue development until the first time you tried to debug the run, and only then it would state that that is not the supported way to use the components. And then you would need to refactor the whole thing again. Now you can’t really use the components in the way they are not supported. Also one welcomed improvement: default timeout was changed from 7 days to 12 hours.
- Synapse connectivity improved – Synapse Link for SQL
The feature offers near real-time data movement from SQL (Azure SQL Database or SQL Server 2022) to Synapse Dedicated pool. That is great for fast reporting needs. You can even have some daily operational reporting done via your data warehousing environment as you can see the “current” – almost live – situation on reports when configured correctly. Of course, this doesn’t remove the need for the actual reporting models and integrations of a data warehousing environment, but gives some rope on how often the data can be refreshed.
Things to look forward to (2023) – Azure Data Platform
- Virtual network (VNet) Data Gateway
Feature still in preview and only available for Power BI Premium version. Data gateway created a Secure connection to VNET and removes the need for On-Prem gateway when connecting to resources on private VNET. This makes connecting a lot easier and removes most of the administrative work needed to keep the gateway running.
- SQL DB procedure REST API
Triggering basically any REST API from your Azure SQL DB is now possible and this offers so many ways of using procedures etc. There is a list on the documentation page that states all the allowed endpoints, but you can also expose your own API. Of course, you can use function app or something like Power Automate runs that are exposed as REST.
If you want to invoke a REST service that is not within the allowed list, you can use API Management to securely expose the desired service and make it available to sp_invoke_external_rest_endpoint
- Time-To-Live (TTL) in managed virtual network
This feature is still in preview, but it is a must-have for everyone using the Managed Virtual Networks in Synapse. Before this feature, resources had to be fetched for each call within the loop separately. Now, Time-To-Live feature can be used to assign resource a lifetime that determines how long it will be reused for. The benefit of this is big when, for example, copying functions are run in a large loop, and the resource needed for copying is not always recreated, but an existing one is used.
Naturally, Microsoft is constantly developing its solutions, and more and more features are being added all the time. So we stay alert and always ready to learn, and for a good reason – the benefits they bring can be big. 2022 was a great year in this context, and we can’t wait for the next year.
Best regards,
Mikko and Ilkka
Ps. More exciting things are coming up next year from us, so stay tuned!
Pps. Below some enthusiastic vibes from Microsoft’s Partner Architects Day. Best thing overall about 2022 was to be able to meet and greet people in real-life.
#IsletGroup #data #analytics #PowerPlatform #DataPlatform #Microsoft #reporting