Data Store

Image from https://www.pexels.com/@tiger-lily/
Image from https://www.pexels.com/@tiger-lily/

Cloud providers offer different storage options catering to different needs. In this blog, we look at 3 data storage options provided by Azure. They offer the following

  1. Storage with Query Engine (Relational)
  2. Storage with Query Engine (Non-SQL)
  3. Storage without/limited query support

1. Storage with Query Engine (Relational)

This is a cloud-based database service where the data is organized in rows and columns, and the schema for every table is well-defined. Azure SQL Server offers different deployment options.
  1. SQL Server on Azure VM (IaaS)
  2. Azure SQL Database (PaaS)
  3. Azure SQL Managed Instance (PaaS)
I shall not attempt to dig deep into these options because many online resources are available. We choose this service because
  1. The data model is relational and well-defined
  2. high availability
  3. SQL supports
  4. Fast response time
These are the cost considerations
  1. Deployment model
    1. Single Database deployed to an Azure VM
    2. Elastic Pool of databases, 
    3. Managed Instance (fully-managed SQL Database instance)
  2. Service tier
    1. General Purpose
    2. Business Critical
  3. Computing model
    1. Dedicated resources
    2. Serverless (bill based on usage)
  4. Computing power (number of vCores)
  5. Storage space
    1. data in GB per month
    2. with or without backup options

2. Storage with Query Engine (Non-SQL)

Azure ComosDB offers different data models such as JSON and Graph (Network). We can use SQL for JSON data models and Gremlin on Graph. We choose this service because
  1. The data model is non-relational
  2. high availability
  3. SQL for JSON and Gremlin on Graph
  4. Fast response time

On the billing aspect, we are paying for storage and request units (RUs). There is a calculator for cost estimation.


3. Storage without/limited query support

Azure Blob Storage is a good option when we just want to store data and retrieve them without much query support. For instance, we store computation results, data for machine learning, temporary data, and configuration data in blob storage. Essentially, we are just paying for storage space.

Data is organized in blob containers and blob blocks. A storage account can contain an unlimited number of containers, and a container can store an unlimited number of blobs.

Azure Blob Storage is a good choice for big data solutions, because of its high availability, and low cost. Among other things, it provides hot, cool, and archive storage tiers for different use cases.


Conclusion

Depending on our business needs, we choose the appropriate data store. In this manner, we reduce costs for our cloud solutions and get the most out of these storage options.



Comments

Post a Comment

Popular posts from this blog

Happy New Year, 2024 from DALL-E

Image
generated by DALL-E Wishing everyone a happy new year. Year 2023 is great for me because I learned much about OpenAI. I am involved in a live deployment with it, and involved in a handful of interesting user scenarios around it. On top of this, a copilot hackathon. The image above is generated by DALL-E and following is the code. 1. Dependencies python = "^3.10" pydantic-settings = "^2.1.0" openai = "^1.3.8" 2. Environment Parameters OPENAI_AZURE_ENDPOINT="<your azure openai endpoint>" AZURE_OPENAI_API_KEY="<your azure openai secret>" OPENAI_API_VERSION="2023-12-01-preview" OPENAI_DALLE_MODEL="dalle" Note: OPENAI_API_VERSION has to be 2023-12-01-preview otherwise you will get an exception openai.NotFoundError: Error code: 404 - {'error': {'code': '404', 'message': 'Resource not found'}} dalle  is the name of my dall-e-3 deployment model. You need to change it to
Read more

Bing Search as tool for OpenAI

Image
  Image from https://www.pexels.com/@skitterphoto/ Many times, we need to search for information in the web when we are working with OpenAI. One of the reasons is that the dataset for training for OpenAI is never up to date. For instance, the OpenAI GPT-4 Turbo has knowledge of events up to April 2023. Here is the Python implementation with Pydantic model. Dependencies python = "^3.10" pydantic-settings = "^2.1.0" pydantic = "^2.5.2" azure-cognitiveservices-search-websearch = "^2.0.0" Environment These are the environment parameters needed. Have the following in a .env file. azure_bing_search_endpoint="https://api.bing.microsoft.com" azure_bing_search_key="<key>" We look under "Keys and Endpoint" for these values. Pydantic Setting Model from pydantic_settings import BaseSettings class BingSearchSettings(BaseSettings): azure_bing_search_endpoint: str azure_bing_search_key: str class Config:
Read more

OpenAI: Functions Feature in 2023-07-01-preview API version

Image
  Image from https://www.pexels.com/@pavel-danilyuk/ One of the cool features of OpenAI version 2023-07-01 is its custom Functions feature. This feature is close to me because I am working on a project where we require OpenAI to generate JSON responses. We always want the generated JSON to be consistent, otherwise, we need to do some post-processing tasks to get the JSON object in the correct schema. In the blog, we observe the completion with and without this new feature. Development Setup python -m venv .venv poetry init poetry shell poetry add openai poetry add python-dotenv poetry add pydantic I am using Python version 3.11 Create .env file OPENAI_API_TYPE="azure" OPENAI_API_BASE="<OpenAI endpoint>" OPENAI_API_KEY="<OpenAI secret>" OPENAI_API_VERSION="2023-07-01-preview" OPENAI_DEPLOYMENT_ID="<deployment identifier>" Without OpenAI functions feature We use two sample texts for experimentation. We want to get the na
Read more