K-Means Python script to Flask application (step by step) - PART #1 @DataScienceGarage

Data Science Garage This tutorial demonstrates how to build a stylish #Flask application from a single Python code executing K-means clustering (unsupervised learning). The main idea of that is we need to have centroids of clusters generated by K-means saved to be used by Flask application later. The input value for specified features entered directly in web browser window via built Flask application, and then send to algorithm which calculates the Euclidean distance between entered values and fixed centroid positions. The predicted cluster is that which has the shortest distance between entered values and K-means centroids. Be noticed that fixed cluster centroids from K-Means are saved in a simple lightweight pickle (.pkl) file locally.

At the end of this tutorial, you will have a stylish web application which allows you to enter your values for the algorithm and get results in a second just after you click Submit button at the main page.

I suppose you are little (at least) familiar with HTML in web developing (required) and CSS (not required). If not, this is not difficult to build a nice app. Just by following steps in this video.

The content of the tutorial is:
0:00 – Introduction and DEMO
0:48 – Step #1. Python file for K-Means clustering
2:15 – Step #2. Prepare your project structure
4:25 – Step #3. Write your Flask application
11:45 – Step #4. Prepare a Landing page on index.html
19:08 – Step #5. Have your logo!
19:30 – Step #6. Prepare HTML page for results
20:50 – Step #7. Set-up your web application style with CSS
22:35 – Step #8. Test application and debug
24:16 – Final Flask application for clustering
25:36 – Bonus: Github repository for the project

Used resources:
- Fonts: Google Fonts from fonts.google.com
- Flask documentation (official): flask.palletsprojects.com/en/2.0.x
- Github repo for the project: github.com/vb100/KMeans-FlaskApp

Once you have a working Flask based application locally, you can deploy it in the cloud provider that the application would be public and open worldwide, by using Docker, AWS, Azure, GCP, or other solution upon your business requirements.

Link to the Part 2: Deploy Flask application to AWS Beanstalk as a web app. (youtu.be/fGxY_Hji8_U)

Thanks for watching, hope it was useful!
#kmeans #centroids

updated 4 years ago

K-Means Python script to Flask application (step by step) - PART #1

Data Science Garage 2022-03-10 | This tutorial demonstrates how to build a stylish #Flask application from a single Python code executing K-means clustering (unsupervised learning). The main idea of that is we need to have centroids of clusters generated by K-means saved to be used by Flask application later. The input value for specified features entered directly in web browser window via built Flask application, and then send to algorithm which calculates the Euclidean distance between entered values and fixed centroid positions. The predicted cluster is that which has the shortest distance between entered values and K-means centroids. Be noticed that fixed cluster centroids from K-Means are saved in a simple lightweight pickle (.pkl) file locally.

At the end of this tutorial, you will have a stylish web application which allows you to enter your values for the algorithm and get results in a second just after you click Submit button at the main page.

I suppose you are little (at least) familiar with HTML in web developing (required) and CSS (not required). If not, this is not difficult to build a nice app. Just by following steps in this video.

The content of the tutorial is:
0:00 – Introduction and DEMO
0:48 – Step #1. Python file for K-Means clustering
2:15 – Step #2. Prepare your project structure
4:25 – Step #3. Write your Flask application
11:45 – Step #4. Prepare a Landing page on index.html
19:08 – Step #5. Have your logo!
19:30 – Step #6. Prepare HTML page for results
20:50 – Step #7. Set-up your web application style with CSS
22:35 – Step #8. Test application and debug
24:16 – Final Flask application for clustering
25:36 – Bonus: Github repository for the project

Used resources:
- Fonts: Google Fonts from fonts.google.com
- Flask documentation (official): flask.palletsprojects.com/en/2.0.x
- Github repo for the project: github.com/vb100/KMeans-FlaskApp

Once you have a working Flask based application locally, you can deploy it in the cloud provider that the application would be public and open worldwide, by using Docker, AWS, Azure, GCP, or other solution upon your business requirements.

Link to the Part 2: Deploy Flask application to AWS Beanstalk as a web app. (youtu.be/fGxY_Hji8_U)

Thanks for watching, hope it was useful!
#kmeans #centroids

How to run LLM locally with ollama | Python example

Data Science Garage 2024-06-18 | My colleagues from IBM advised to try ollama, because this tool is super useful if you want to run and use LLM models locally. Just download the LLM model quantized GGUF file with ollama, prepare Python file for inference, custom it based on your needs and use!

While recording this video I have never had experience with ollama before. So, I demonstrated very basics of ollama, including the usage of website and enhancing the Python file from ollama repo to have direct inference to local server LLM model.

In this example I used llama3 and llama3-chatqa (compatible with RAG) LLM models. This requires to have ollama CLI (command-line interface) installed on your PC. All steps are demonstrated within the tutorial.

About GGUD format:
GGUF is a binary format that is designed for fast loading and saving of models, and for ease of reading. Models are traditionally developed using PyTorch or another framework, and then converted to GGUF for use in GGML. This is like compressed model version in terms of size on disk.

I definitely will use ollama for a series of LLM-based tasks and experiments in future. Do you have your own experience on that? Let me know in the comments! Thanks!

- - - - - - - -
References mentioned and showed in the video:
- ollama website: ollama.com
- Simple Generate Example with Python: github.com/ollama/ollama/tree/main/examples/python-simplegenerate
- GGUF format: huggingface.co/docs/hub/gguf

#ollama #huggingface #llm

- - - - - - - -
Content of the video:
0:00 - Intro
0:33 - ollama website
1:27 - Download ollama CLI to Mac
2:18 - Test ollama CLI first time with llama3 (pull the model)
3:24 - Take llama3-chatqa model for the 2nd experiment
4:03 - Download and run llama3-chatqa model locally
5:27 - Using ollama Github repo
5:55 - Simple Generate with Python example
7:04 - Check ollama ports for running locally
7:38 - Test llama3-chatqa LLM model with CLI
8:11 - Final word

Configuration parameters for LLMs | Clearly Explained

Data Science Garage 2024-04-08 | Want to optimize LLMs performance for your specific needs? In this video I will unravel the importance of adjusting configuration parameters, such as Random sampling, Greedy samply, as well as Top K and Top P sampling strategies to develop your own powerful LLM.

With Top K and Top P (Random sampling) you can easily control, how your LLM model should be creative or pragmatic.

Temperature and Max New Tokens are crucial parameters in developing LLMs because they directly influence the quality, coherence, and diversity of the generated text or next word/token generation.

All of these and more are clearly explained in this video, where every parameter is visualized and represented with examples.

By delving into these configuration parameters, you'll gain the knowledge and skills needed to fine-tune your LLM for various tasks, from generating creative stories to aiding in natural language processing tasks. Join us on this journey to enhance your understanding of LLMs and optimize their performance.

Remember that attention mechanism is the core of LLM archirecture. Because the softmax layer in the LLM calculates the probability distribution over the input tokens, indicating their relative importance for generating the next token.

The content of the video:
0:00 - Intro
1:10 - Attention mechanism for LLM
1:36 - Max New Tokens parameter
2:33 - Greedy vs. Random Sampling
4:56 - Top K parameter
5:59 - Top P parameter
6:51 - Summary of Top K vs. Top P
7:10 - Temperature parameter for LLMs
8:03 - The effect of low temperature for next token generation
9:05 - The effect of high temperature for next token generation
9:43 - The default temperature value in LLM

In case of any comments or suggestions, let me know in the comments below!

#LLM #largelanguagemodels #finetuning

Get LLM output as Python object with Langchain and Pydantic | Hands-on tutorial

Data Science Garage 2023-12-21 | The #Pydantic output parser is a tool that allows users to define a JSON schema to query LLMs for outputs that adhere to that schema. This is pivotal for applications that require structured data, as it ensures outputs conform to predefined formats. The parser leverages Pydantic’s BaseModel for data validation and type checking.

With this tutorial you will learn to: set up the data model in a way that Langchain’s output parser can be used to generate structured data.

For example, you can grab from #LLM not a plain text (as an answer), but re-usable Python objects, such as Python list, dictionary, Pandas dataframe and more.

This functionality allows you to create super powerful LLM applications where any kind of data transformation, parse or passing to ML models (such as example) are required.

To utilize this parser, one must define the data structure using Pydantic’s BaseModel. You will learn that in the tutorial.

Useful links and references:
- Pydantic (JSON) Parser: python.langchain.com/docs/modules/model_io/output_parsers/pydantic
- PydanticOutputParser: api.python.langchain.com/en/latest/output_parsers/langchain.output_parsers.pydantic.PydanticOutputParser.html
- LLMs supported by LangChain: python.langchain.com/docs/modules/model_io/llms

In this tutorial I used OpenAI's ChatGPT API to support LLM. Feel free to use any supported LLM using LangChain (now testing on IBM WatsonX).

LInk to Github repository with the original code used in this tutorial: github.com/vb100/langchain_pydantic

The content of the tutorial:
0:00 - Main idea using Pydantic with Langchain
1:09 - Implementation scheme for hands-on
3:29 - Hands-on part (coding)
15:06 - BONUS: Github repo

#langchain

Happy learning!

SequentialChain with LangChain - Python Example (explained)

Data Science Garage 2023-10-26 | Sequential chains consist of steps where each step takes one input and produces one output. It is one of the most popular #LLM variant in #LangChain.

In simple words, LangChain is a framework that simplifies the process of creating generative AI application interfaces basically in within the NLP context.

In Sequential chains, the output from one step becomes the input for the next. You can simply setup individual chains, define input and output for them and combine them all into logical structure with LangChain.

This tutorial covers both some theory and hands-on practice, so it can be useful either for beginner and advanced user in Large Language Models developing. The whole code is written in purely Python.

As the main LLM framework we used here in the video, is OpenAI's gpt-3.5-turbo (the current version of #chatgpt ). With this video we will develop a logical sequential chain which takes an input the employee's performance review, and delivers the output as personalized plan forr improvement based on summarized employee's weaknesses. That is almost real-life example you go deeper with this tutorial

The content of the video:
0:00 - Intro
0:23 - Idea of the example
2:41 - Hands-on with Python
12:08 - Explaining results

After this tutorial you will be more familiar with prompt engineering using LLM templates (instructions what to do for individual chains), defining templates for LLM and designing custom chains for you business use case.

Useful links:
- Official documentation about LangChain Sequential Chains: python.langchain.com/docs/modules/chains/foundational/sequential_chains
- Full Python code and input data at Github repo: github.com/vb100/langchain_sequentialchain_demo

Enjoy and have fun!

Get Started with LangChain LLM Models - Step by Step with Python

Data Science Garage 2023-10-14 | If you are new to LLM and want to start you learning, the best tool is LangChain which is super easy to use with Python. Basically, LangChain is a framework built around LLMs. We can use it for chatbots, Generative Question-Answering (GQA), summarization, and much more. The core idea of the library is that we can “chain” together different components to create more advanced use cases around LLMs.

This video can be your kick-start to your journey into Generative AI or Large Language Models from coding perspective. With this tutorial you will be able to write questions, queries or other requests to OpenAI, to be more specific, to ChatGPT chat models.

I think, as a good start is LangChain's generation function, which generate random outputs to your inputs within Python code.

With OpenAI.generate method you will be able to construct and manage multiple queries at once.

The content of the tutorial:
0:00 - Intro
0:26 - Prepare Python virtual environment and install LangChain
1:00 - Setup OpenAI API Key
2:10 - Load OpenAI API Key to Python code
2:56 - Install OpenAI Python package
3:14 - LM Object Initialization with LangChain
4:17 - Simple LLM usage example and increase OpenAPI quote
5:59 - Use generations in LLMs with LangChain
8:47 - Final word

Note: text-davinci-003 text completion model was used when the video was recorded.

References to material mentioned in the video:
- LangChain installation: python.langchain.com/docs/get_started/installation
- OpenAI quote: platform.openai.com/account/billing/limits

If you want to get more GenerativeAI, Data Science, Python programming or any new related high quality content, please subscribe the channel @DataScienceGarage !

#langchain #python #LLM

Build Custom Text Classification Model with Only Few Sample | Spacy and Setfit

Data Science Garage 2023-07-07 | You can train state-of-the-art text classification models with only a few samples straight in SpaCy now. Together with Spacy SetFit which is a new few-shot text classification library you can do that very easily just by following steps in this hands-on tutorial.

The combination of Spacy and Setfit allows you to add your small training set (few text samples) that will be used for fine tune the base Spacy model. For example, as in this tutorial, the new text will be classified to two classes: inlier and outlier.

As a base NLP model, we use Universal English Language model (small version), which is named as en_core_web_sm. You can read more on this here: spacy.io/models/en

Also, we use sentence-transformers model released by HuggingFace: paraphrase-MiniLM-L3-v2, which maps our text into tokens. You can find more information on this from here: huggingface.co/sentence-transformers/paraphrase-MiniLM-L3-v2

The Github repository with the full Python code for the tutorial is available here: github.com/vb100/spacy_text_classificator

Subscribe the @DataScienceGarage channel to get more high quality tutorials, reviews and explainable videos!

- - -
If you want to change you career and became advanced data analytic or data scientist, check this awesome Turing College!
Meet industry leaders and take your role in the job market with heavy baggage of you skills!
Visit: turingcollege.org/DataScienceGarage !

---
The content of the tutorial:
0:00 - Intro
0:27 - Install Spacy and Spacy Setfit
1:00 - Install en_core_web_sm
1:33 - Setup a Python file to implement text classification
6:16 - Test the fine-tuned NLP model on test data
7:41 - Bonus: Github repository and the best data science school

#nlp #python #setfit #spacy #textclassification

Data Scientist Role | My Roadmap

Data Science Garage 2023-05-25 | I am data scientist at IBM, where is a great place to work with different AI, Machine Learning and Data Science projects around the globe. It is significant experience. However, there is a long and tiring road to this behind.
This video introduces my long and challenging way to become firstly Data Analytic, and then how I jumped into Data Scientist role.

Also, this video introduces I think the best place to learn Data Analytic and Data Science online with the best industry leaders where you will get job-industry based study plans designed based on real job market requirements. Additionally, perfectly organized educational system and industry professionals will lead you through your tasks and business-based projects.

Turing College is rezognized education school where you will learn as in real college, with tasks, projects, weekly meetings and even more. Turing College has strong partnership with big IT industry companies where you can be recommended to take your first Data Scientist or Data Analytic role.

With the Turing College, you will get not only learn technical and soft skills, but also Senior team leads, mentors and industry leaders will share to you career advices and recommendations that probably you not gonna give anywhere else. Due to this educational packet you will easy plan you data scientist career in case you put your energy and efforts into learning, communicating and sharing your experience.

Need more details, or want to join? Check HERE to learn more: turingcollege.org/DataScienceGarage

See you in the college!

#datascience #turingcollege #datasciencecourse

- - -
Video sections:
0:00 - My story how I became data scientist at IBM
5:36 - Why Turing College
6:05 - Requirements for good data science school
8:14 - Study plans
9:48 - Learning workflow at Turing College
12:27 - Data Science Master degree

Polars vs Pandas | detailed test with explained results

Data Science Garage 2023-04-26 | Polars is one of the most trending Python framework which hits Pandas in many performance tests. This video presents 8 distinct tests which demonstrates differences between Pandas and Polars in duration in seconds while running specific functions on data.

I tested the following functions by scoring these two frameworks:
- Test 1: read a single CSV file
- Test 2, and 3: select columns from a loaded dataframe (two approaches).
- Test 4: Filtering data in a dataframe.
- Test 5 and 6: Create a new column (two approaches).
- Test 7: Group and aggregate data.
- Test 8: Fill missing data.

I evaluated the competition in two groups:
1. Group where I did not used Lazy evaluation in Polars.
2. Group where I used Lazy evaluation in Polars.

From a high level perspective, Polars represents data in memory with Arrow arrays while Pandas represents data in memory in Numpy arrays. For this reason, Polars suggest Lazy functionality which makes it much faster. I mentioned it multiple times in this video (Polars has Eager and Lazy APIs, while Pandas can suggest Eager only).

The content of the whole experiment is:
0:00 - Intro
1:08 - Introducing experiment Python code
12:05 - Run the experiment
18:04 - Experiment results (summary).
21:22 - Final test results.

- Github repo with Python codes used in this experiment: github.com/vb100/polars_vs_pandas

Additional material:
- Official Polars documentation: https://www.pola.rs/
- Lazy functionality in Polars: towardsdatascience.com/understanding-lazy-evaluation-in-polars-b85ccb864d0c

#polars #pandas #experiment

20 AI tools that makes your life easier | Review

Data Science Garage 2023-04-06 | AI innovation is moving at lightning speed. Every day new AI and ML products are releasing. ChatGPT and GPT-4 can be interpreted as the intelligence of new era. Significant portion of AI products introduced in this video uses these new technologies to make your life easier in various areas and activities.

Here are 20 of the best AI tools I've found recently and want to share with you:
0:00 - Intro
0:16 - questAI: Let's you build web applications faster.
0:52 - Alan AI: Helps you add an AI voice assistant to your applications.
1:31 - D-ID: An AI tool to create digital avatars giving you an immense human-like conversational experience.
2:14 - Figma AI: Designing tool that helps you create websites, applications, logos and much more.
2:55 - Magical AI: Helps you automate repetitive tasks and saves hours per day.
3:35 - Remove.bg: Remove background from any image using AI.
4:16 - Sidekick AI: Helps you to schedule your meetings.
4:57 - Hints AI: Digital organizer to manage and organise your day-to-day activities.
5:34 - Publer: Tool to create, schedule or analyse all social posts on one platform.
6:20 - AI Image Enlarger: Enlarge and upscale your images without losing quality.
6:58 - Markopolo AI: Helps you do digital advertising on autopilot.
7:36 - Murf AI: Make studio quality voice overs from input text in seconds.
8:22 - Longshot AI: Writing assistant for accurate and optimised content writing.
8:59 - AI Picasso: Let's you create amazing artwork using AI.
9:33 - Rephrase AI: Use it to convert you boring text into highly engaging videos in minutes.
10:13 - Delv AI: Helps you learn faster by retrieving summarised data from any text file.
10:54 - Durable AI Website Builder: Make amazing websites in 30 seconds from scratch.
11:39 - There's is an AI for That: An inventory to find new AI tools for any task.
12:19 - Hypotenuse AI: Create unique copywriting and visual content using AI.
12:56 - Browse AI: Let's you scrape any website within 2 minutes.
13:31 - Outro

Also, there are many other AI-based tools which shaking the industry, and maybe tomorrow the list would be different. But, what do you think about these AI tools? Did you tried it?

Drop a comment and share your experience!

Thank you for watching!
Vytautas - @DataScienceGarage

#gpt4 #chatgpt #aitools

Load data from AWS S3 to Snowflake Worksheet with SQL

Data Science Garage 2023-03-29 | This tutorial demonstrates how to efficiently load and query data from AWS S3 bucket to your Snowflake worksheet. This video is dedicated to Snowflake beginners who want to try Snowflake for the first time and has no any previously experience on this technology.

Main take-aways from this tutorial, you will learn how to:
- create new Snowflake worksheet.
- create and understand what is staging area (staging table).
- write SQL queries in Snowflake worksheet.
- create IAM user in AWS and assign S3 Full access privileges.
- get and use IAM user credentials (AWS ACCESS ID and AWS SECRET KEY)
- load data from S3 bucket to Snowflake SQL table.

From the data science perspective, Snowflake was designed from the ground-up to support Machine Learning (ML) and AI-driven data science applications.

With Snowflake you can build apps, data pipelines and design ML workflows.
Official website for developers: developers.snowflake.com

In conjunction with tight integration to Spark, R, Python and other programming languages and frameworks, Snowflake is indispensable data science technology.
One of the main reason is that performance speed is a key factor in supporting robust ML models.

Additional features that Snowflake supports:
- SQL API.
- External tables.
- Iceberg tables.
- External functions.
- SQL stored procedures.
- Native app framework.
- Continuous data pipelines.
- Streams and tasks.
- Separation of compute and storage.
- Workload isolation.
- Semi-structured data.
- Unstructured data.
- Connectors and drivers.
- Time travel (easily restore and analyze deleted or modified historical data without backups).
- Access controls and encryption.
You can check for all features here: docs.snowflake.com/en

Happy learning!

#snowflake #aws #sql

SQL with Python with Polars DataFrame library (hands-on tutorial)

Data Science Garage 2023-03-03 | This video shows how to execute SQL queries with Python with Polars DataFrame library. You can use Polars for any kind of tabular data stored in CSV, Parquet, or other standard data file formats.

Polars module is built with Rust which gives it C++ performance and allows to control performance. There are two API:
- for Rust user: pola-rs.github.io/polars/polars
- for Python user: pola-rs.github.io/polars/py-polars/html/reference

You can use Polars as a DataFrame library or as query engine backend for your data models. Because of this reason it is beautiful choice for data scientists and data analytics who need handle big amount of data or are more familiar with SQL than Pandas (in Python).

Polars supports Numpy universal and Windows functions, also provides so popular statistics and aggregation functions such as GroupBy, Folds, and Regular Expressions. Also, you can use it with Selecting, Handling, Combining, Multiprocessing data, and even with Time Series data.

Useful links
- Polars User Guide: pola-rs.github.io/polars-book/user-guide/index.html
- Polars main website: https://www.pola.rs/

Content of the video:
0:00 - What is Polars DataFrame library
1:32 - Your first Python code with Polars

In some aspects, Polars can be a good alternative to Spark SQL framework.

If you found useful from this tutorial, please drop a comment or subscribe to get more similar videos!
@DataScienceGarage

#sql #polars #rust #python

Testing #OpenAI API (GPT3 and other) with Python

Data Science Garage 2023-01-30 | Use #OpenAI API with Python and test 3 different use cases, including text completion with #gpt3, code completion with #codex, and image generation with #dalle . It is very easy to do, as I demonstrated in the full video here: youtu.be/yNSo6cWpDSM

Link to Github repo with Python scripts (examples for all three tasks): github.com/vb100/openai-test-api

OpenAI API with Python | 3 Use Cases with codes

Data Science Garage 2023-01-29 | OpenAI API allows you to access GPT-3, which performs a variety of natural language tasks, Codex, which translates natural language to code, and DALL-E, which creates and edits original images. Today you can test OpenAI API for free, and that is fantastic opportunity to play with AI.

This video suggest you to practically try all this asset directly on you machine. All what you need to have is a openai Python package, which you can install from official PyPi repository: pypi.org/project/openai

This video demonstrates real hands-on with Python codes which you can try yourself and provides few different results for each use case.

The content of the video:
0:00 - When ChapGPT API will be released?
0:44 - Generate OpenAI API key
1:04 - Install OpenAI for Python
1:27 - Use case 1. Text Completion
3:31 - Use case 2. Code Completion with Codex
5:40 - Use case 3. Image Generation with DALLE-E

Additionally, you can use Beta OpenAI Playground to test natural language models released by OpenAI directly in a pre-defined workspace: http://beta.openai.com/playground

- - -
Links mentioned or appeared in the video:
- (Article): ChatGPT moves to cash in on its fame as OpenAI launches plan to charge monthly fee for premium subscribers - fortune.com/2023/01/23/chatgpt-openai-plan-monthly-fee-for-premium-subscribers
- (Article): The ChatGPT API from OpenAI is on the way, and it will also be available in Microsoft Azure: This means… (wishmatv.com/technology/the-chatgpt-api-from-openai-is-on-the-way)
- (OpenAI Community): Is there an API for ChatGPT3? - community.openai.com/t/is-there-an-api-for-chatgpt3/23871
- (Article): OpenAI’s ChatGPT will be available as an API soon - techinformed.com/openais-chatgpt-will-be-available-as-an-api-soon

Github repo with Python codes (3 files for each use case): github.com/vb100/openai-test-api

- - -
Documentation for OpenAI use cases mentioned in the video:
1. Text completion: beta.openai.com/docs/guides/completion
2. Code completion: beta.openai.com/docs/guides/code/best-practices
3. Image Generation: beta.openai.com/docs/guides/images/usage

Thank you for watching!
Vytautas - @DataScienceGarage

#openai #chatgpt #imagegeneration

OpenAI Whisper - Fine tune to Lithuanian | step-by-step with Python

Data Science Garage 2023-01-11 | Fine-tune OpenAI's Whisper to different language is simple using Python and Google Colab with GPU. In this tutorial, I selected the small version of Whisper AI model to fine-tune to Lithuanian language. Whisper can transcribe 96 other languages along with also being able to translate from those languages to English

Also, this video partly explains Whisper AI paper (tokenizer, encoder, decoder, padding, and other) and the model itself.

Before starting hands-on with Whisper, you should create your Hugging Face token at: huggingface.co/settings/tokens

You can check language dataset from Mozilla-foundation used in this tutorial at: huggingface.co/datasets/mozilla-foundation/common_voice_11_0

Using Whisper for transcription in Python is very easy.

Whisper is an automatic speech recognition (ASR) system released by OpenAI and trained on 680000 hours of multilingual and multitask supervised data collected from the web. The Whisper architecture is a simple end-to-end approach, implemented as an encoder-decoder Transformer. Input audio is split into 30-second chunks, converted into a log-Mel spectrogram, and then passed into an encoder. A decoder is trained to predict the corresponding text caption, intermixed with special tokens that direct the single model to perform tasks such as language identification, phrase-level timestamps, multilingual speech transcription, and to-English speech translation.

For this video example, we will use a small version of WhisperAI model.
You can check for all available versions in the official Whisper AI Github model card: github.com/openai/whisper/blob/main/model-card.md

The sections are:
0:00 – Hands-on steps
3:14 – Install PyTorch for WhisperAI with CUDA
3:34 – Set GPU Runtime in Google Colab
4:14 – Install ffmpeg package on the machine
4:40 – Install dependencies for fine-tuning
5:35 – Step 0. Log in to Hugging Face
6:09 – Step 1. Loading the dataset
7:16 – Step 2. Prepare Feature Extractor and Tokenizer
8:24 – Step 3. Combine elements with WhisperProcessor
9:06 – Step 4. Preapare data
11:03 – Step 5. Training and Evaluation
11:09 – Step 5.1. Initialize the data collator
12:26 – Step 5.2. Define evaluation metrics
12:56 – Step 5.3. Load a pre-trained Checkpoint
14:13 – Step 5.4. Define the training configuration
15:48 – Step 5.5. Train the Whisper AI model (fine-tune)

The Github repo with the full code available at: github.com/vb100/whisper_ai_finetune

Technical definitions mentioned in the video:
- WhisperFeatureExtractor: huggingface.co/docs/transformers/model_doc/whisper#transformers.WhisperFeatureExtractor
- WhisperTokenizer: huggingface.co/docs/transformers/v4.24.0/en/model_doc/whisper#transformers.WhisperTokenizer
- WhisperProcessor: huggingface.co/docs/transformers/v4.24.0/en/model_doc/whisper#transformers.WhisperProcessor
- WhisperForConditionalGeneration: huggingface.co/docs/transformers/model_doc/whisper#transformers.WhisperForConditionalGeneration
- LogMel Spectogram: medium.com/analytics-vidhya/understanding-the-mel-spectrogram-fca2afa2ce53

@DataScienceGarage - subscribe and get more high quality content soon!

#whisperai #openai #transcription

Train logistic regression model with SparkML

Data Science Garage 2022-12-12 | How to build and train a binary classification model with Spark (SparkML with Python).
Full hands-on tutorial with theory and practice. Github repo included.

LINK TO FULL TUTORIAL ON YOUTUBE: youtu.be/7-C2ikVNInQ
LINK TO GITHUB REPO: github.com/vb100/spark_ml_train_model

#datascience #ai #machinelearning #spark #python #sparkml #pyspark #statistics #tutorial #ml #mlops

Train Machine Learning Model with SparkML (...and Python) | Hands-on tutorial

Data Science Garage 2022-12-06 | To build and train a Machine Learning (#ML) model with Spark is not hard. With this tutorial we will build a simple Binary Classification ML model with Spark. We will use Logistic Regression built-in Spark algorithm, and then evaluate it by getting performance metrics from the model.

There are some different from we do it in Scikit-Learn. Spark provides a built-in SparkML engine with rich #SparkML API which you can leverage to build your unique Machine Learning model.

In this tutorial we are using SparkUI v.3.2.1 with pyspark-shell.

The critical points you should pay your attention to is:
- Datatypes (DTypes)
- String Indexer and One-Hot-Encoding for categorical features.
- Vector Assembler.

All these parts are explained and demonstrated in details in this tutorial. Also, you will learn what is SparkContext and SparkSession (differences between them). Therefore you will be able to check Data schema and handle data types in Spark DataFrame, selected features within your data. As required for ML modelling, you will also learn how to split your data into train and test sets.

Here you also learn how to setup ML stages with Spark and build a custom ML Pipeline to build your Machine Learning Model with Spark.

At the end, you will learn hot to get model performance metrics, such as Precision, Recall, or ROC curve values.

The tutorial is prepared with Jupyter Notebook, using Python programming language, so all the steps are executed with #pyspark .

The content of the video:
0:00 - Intro
0:32 - Start of Hands-on with Jupyter Notebook
0:46 - 1. Import main dependencies for Spark and Python
1:14 - Theory: Spark Session vs. Spark Context
3:10 - 1. Continuing importing dependencies
3:28 - 2. Load External CSV data to Spark (as Spark DataFrame)
5:40 - 3. Train and Test splits
6:39 - 4. Check Data Types
8:27 - 5. One-Hot-Encoding with Spark
10:07 - Theory: StringIndexer and One-Hot-Encoer
11:01 - 5. Continuing with StringIndexer hands-on
12:19 - 6. Vector Assembling
12:55 - Theory: Vector Assembling in Spark
13:53 - 6. Continuing with Vector Assembling
15:24 - 7. Make Spark ML Pipeline
18:31 - 8. Train ML Model with Spark
20:07 - 9. Get Model Performance Metrics

Spark API and SparkML API method used in the tutorial (incl. documentation):
- Spark Datatypes (spark.apache.org/docs/latest/sql-ref-datatypes.html)
- PySpark SQL DataFrame Random Split (spark.apache.org/docs/3.1.3/api/python/reference/api/pyspark.sql.DataFrame.randomSplit.html)
- StringIndexer (spark.apache.org/docs/latest/api/python/reference/api/pyspark.ml.feature.StringIndexer.html)
- OneHotEncoder (spark.apache.org/docs/3.1.1/api/python/reference/api/pyspark.ml.feature.OneHotEncoder.html)
- VectorAssembler (spark.apache.org/docs/latest/ml-features#vectorassembler)
- Spark DataFrame aggregation (spark.apache.org/docs/latest/api/python/reference/pyspark.pandas/api/pyspark.pandas.DataFrame.aggregate.html)
- Count Distinct values from Spark DataFrame (spark.apache.org/docs/3.1.2/api/python/reference/api/pyspark.sql.functions.countDistinct.html)
- Group by to check feature distribution (spark.apache.org/docs/latest/api/python/reference/pyspark.pandas/api/pyspark.pandas.DataFrame.groupby.html)
- SparkML Pipelines (spark.apache.org/docs/latest/ml-pipeline.html)
- Logistic Regression in Spark (spark.apache.org/docs/1.6.1/ml-classification-regression.html#logistic-regression)

Link to the Github repo to hand-on everything on your side (data file is included there): github.com/vb100/spark_ml_train_model

Thank you for watching!

Please subscribe this channel - @DataScienceGarage to get more high-quality videos about #DataScience , #Python , #AI , #MachineLearning , #DeepLearning and much more!

Find Outliers with AutoEncoder - Full Tutorial (Hands-on and Theory)

Data Science Garage 2022-11-05 | This video shows hot to find outliers in data by using AutoEncoders. This approach relies on reconstruction errors calculated by AutoEncoders which can be trained in your Jupyter Notebook with Python programming language very quickly. In this video we will setup a very simple AutoEncoder which consists of Encoder and Decoder.

For this example I used a circle-shaped data distribution, but this approach will work for any shape of data distribution. The main idea is that Encoder compress the input data which results in less dimensions of data and then the Decoder attempts to reconstruct the compressed data back to original data distribution.

These data points which are not outliers has low reconstruction error, while these data points which are far away from dominating data distribution (circle-shape in this example) are identified as anomaly or outliers.

Link to the Github repo with Jupyter Notebook and sample data: github.com/vb100/autoencoder_outliers/blob/main/FindOutliersWithAutoEncoder.ipynb

To setup an autoencoder with encoder and decoder, I used Tensorflow 2.10 in this video tutorial.
During the video I ofter visualize the data to better understand what is going on behind to open Black box of the main idea for you.

Before ingesting the initial input to Decoder part, we must to convert our numerical data representation into Tensorflow Tensor. Tensor is that format which is readable for Artificial Neural Network (ANN) in many architectures (Tensorflow, PyTorch, etc.). By having this, we are ready to train our AutoEncoder. I also suggest to apply numpy method to the tensor to be sure that the data type of our tensor is float (floating number).

To calculate the reconstruction errors for our data points we use the MSE (Mean Squared Error) loss function. It is enough to find outliers (anomalies) in our dataset.

The content of the tutorial:
0:00 - Intro
2:23 - Hand-on with Python: (1) Load dependencies
3:35 - (2): Load data with Pandas
7:21 - (3): Setup AutoEncoder with Tensorflow
12:31 (4) Get reconstruction errors
15:57 (5) - Construct a Pandas Dataframe with the results
17:33 - Automating detecting outliers (theory)

Data outliers is still a hot topic in Data Science and Data Analytics (even in Business Analytics) and make a significant impact to business outcomes calculations, Machine Learning model performance and business-related conclusions and insights. This is the reason I decided to create to create this tutorial.

This approach reminds me PCA method which separates signal from noise. You can learn more about that in this tutorial: youtu.be/FyomruL8Avk

#outliers #anomalydetection #findoutliers #statistics #tensorflow #pandas #decoder #encoder #MSE #meansquarederror #machinelearning #python #numpy #pytorch #automating #tutorial #deeplearning #businessanalytics #datascience

Happy learning!
Your - @DataScienceGarage
Subscribe the channel to get more useful videos. See you there!

Youtube API for Python - step by step

Data Science Garage 2022-10-31 | Hello, now I am working with Youtube API and want to build an Python analytic to analyze youtbe data in a channel level. For this, I prepared this ste-by-step tutorial how to use that Youtube API from totally zero. It is very easy. The video is right here:

youtu.be/TIZRskDMyA4

Additional resources:
- Google API Client for Python (Installation): youtube.com/watch?v=TIZRskDMyA4
- Youtube API samples (incl. Youtube Data Analytics, Youtube Live Streaming API) in Python programming language: github.com/youtube/api-samples
- Google Developers documentation for Youtube Data API v3: developers.google.com/youtube/v3

#youtubeapi #youtubedata #googlecloudplatform #googlecloudconsole #gcp #youtubegithub #youtubedataapi #googleapi #apiclient #api #youtubeshorts #channeldata #datamonitoring #datascience #dataanalytics #datascrapping #datacrawling

How to use Youtube API to extract YT channel data with Python (Step by step)

Data Science Garage 2022-10-30 | This video tutorial shows how to extract statistical data from a Youtube channel by using Youtube API provided by Google in a Python code.
You can get statistical data from any Youtube channel which has an unique username or ID by using Youtube Data API v3, which provides wide capabilities to get such statistics and channel’s metadata as:
- View Counts.
- Video Counts.
- Subscribers Count.
- Hidden Subscribers Count.

The video is divided into 4 main parts: firstly I will demonstrate how to setup Google Cloud Console for the API; then we will quickly run through the documentation and resources; then we will be ready to start some hands-on in Jupyter Notebook where we will write some simple lines of Python code; and finally we will come back to GCP and check the Youtube API Dashboard to monitor its data.

In the hands-on part we will import Google API Client, then declare our connection to the API with generated API key. Then we will write our first request to the API, execute it and print the response directly from Youtube API for a given channel.

The content of the tutorial is:
0:00-0:57 : Intro
0:57-3:10 : Setup Youtube API in Google Cloud Console
3:10-3:52 : Install Google API Client for Python
3:52-5:08 : API Documentation
5:08-11:17 : Hands-on with Jupyter Notebook (Python)
11:17 : Monitor data with Youtube API Dashboard in GCP

To get statistics from youtube channels we will use the Youtube Data API v3. This is an API that provides access to Youtube data, such as videos, playlists, and channels.
Also, in Python environment, we will install official Google API for Python library which supports requests to the API and returns responses. I did not check if the API supports Youtube Shorts which are trending today, but if I test it - I will surely share it with you.

Resources mentioned in the video:
- Google API Client for Python (Installation): pypi.org/project/google-api-python-client
- Youtube API samples (incl. Youtube Data Analytics, Youtube Live Streaming API) in Python programming language: github.com/youtube/api-samples
- Google Developers documentation for Youtube Data API v3: developers.google.com/youtube/v3

Specifically, we will use forUsername parameter to identify an Youtube channel in our requests declared in Python code. This approach valid if there is only one YT channel assigned to that user name. Also, there are other ways hot you can filter channels for your requests, you can check it here: developers.google.com/youtube/v3/docs/channels/list

Subscribe the Data Science Garage channel to get upcoming videos first!

I hope this video and material was useful for you. Thank you for watching!
Your -@DataScienceGarage

#youtubeapi #youtubedata #googlecloudplatform #googlecloudconsole #gcp #youtubegithub #youtubedataapi #googleapi #apiclient #api #youtubeshorts #channeldata #datamonitoring #datascience #dataanalytics #datascrapping #datacrawling

Bi-Directional Attention diff. from Self-Attention

Data Science Garage 2022-10-23 | Learn how Bi-Directional attention works in a high level.

#attention #nlp #bidirectional #tokenizer #bert #selfattention #BiDAF #multihead #python #dotproduct #naturallanguageprocessing

@DataScienceGarage

BI-DIRECTIONAL ATTENTION | Explained in high level

Data Science Garage 2022-10-23 | This video tutorial explains how Bi-Directional Attention works in NLP. This attention mechanism is very similar to Self-attention method which was introduced in the previous video. The main difference is that Bi-Directional attention do not have Masking operation, which Self-attention has.

Also, while self-attention look to the previous words (or tokens) only, the Bi-Directional attention looks to both sides. For this reason this attention called as Bi-Directional.

This video do not cover math for this method. This lesson explain the logic in a high level how Bi-Directional attention works. To implement this method, there are developed Python packages to do it.

Bi-Directional attention is widely used in BERT, which means: Bidirectional Encoder Representation from Transformers.

This is the 3rd video in the mini course about Attention in NLP. Check it out the previous ones:
1. Encoder-Decoder attention and Dot-Product: youtu.be/z5upsjfVU9c
2. Self Attention: youtu.be/pfvYuX0Gjys
3. Bi-Directional Attention (this one).
4. Multi-Head attention (Upcoming).

You can read more about Bi-Directional Attention in the following sources:
- Standford University: Bidirectional Attention Flow with Self-Attention: https://web.stanford.edu/class/archive/cs/cs224n/cs224n.1214/reports/final_reports/report170.pdf
- Medium.com article (BiDAF): towardsdatascience.com/the-definitive-guide-to-bi-directional-attention-flow-d0e96e9e666b

See you! - @DataScienceGarage

#attention #nlp #bidirectional #tokenizer #bert #selfattention #BiDAF #multihead #python #dotproduct #naturallanguageprocessing

Github Action with AWS ECR for you ML project - full tutorial

Data Science Garage 2022-10-22 | Learn how to use Github Actions with AWS ECR (Elastic Container Registry) for you ML project.
This step-by-step tutorial guide you through the full process from the beginning to the final deployment of your ML application image on the Amazon Web Services Elastic Container Registry by using Github Action.

Full tutorial is HERE: youtu.be/yv8-Si5AB3U

This approach is popular among those who use MLOps methods in product development and automation.

Enjoy and happy learning with @DataScienceGarage !

#github #aws #githubactions #ecr #docker #elasticcontainerregistry #developer #machinelearning #ml #datascience #mlops #awsmachinelearning #ai

Self-Attention in NLP | how does it works?

Data Science Garage 2022-10-20 | Learn how attention mechanisms works in NLP with mew videos releasing by @DataScienceGarage.

The series of video leassons cover such topics:
- Encoder decoder attention and dot product: youtu.be/z5upsjfVU9c
- Self attention: youtu.be/pfvYuX0Gjys
- Bi-directional attention (in progress).
- Multi-head attention (in progress).

These lessons will help you understand how different attention methods works in Natural Language Processing (NLP).

Enjoy and see you in videos provided by @DataScienceGarage !

#attention #nlp #dotproduct

SELF-ATTENTION in NLP | How does it works? - Explained

Data Science Garage 2022-10-12 | This video clearly explains how Self-Attention works in NLP (Natural Language Processing). The main idea is calculate relationship between an actual token (word) and previous tokens in the same text sequence.

Self-attention is broadly used in areas such as text summarization and text generation. Self-attention was proposed by researchers at Google Research and Google Brain. It was proposed due to challenges faced by encoder-decoder in dealing with long sequences.

The self-attention model allows inputs to interact with each other (i.e calculate attention of all other inputs). To better understand self-attention mechanism, I suggest to check for material about Transformers in NLP, because: the Transformer model revolutionized the implementation of attention by dispensing of recurrence and convolutions and, alternatively, relying solely on a self-attention mechanism.

You can read more about it here: machinelearningmastery.com/the-transformer-attention-mechanism

As always in attention calculation (as we demonstrated in Decoder-Encoder attention in the last video lesson), we need to calculate a Dot product, which consists of K (Keys) transposed and Q (Queries).

The content of the video:
0:00 - Intro to attention and other mechanisms
0:48 - Principal scheme explanation of Self-Attention
6:40 - Quick example

Learn more in this video right now and enjoy!
I hope the attention intuition introduced in this video let you understand how self-attention mechanism works in more details. If yes, I would be so glad of that.

The first lesson: Encoder-Decoder Attention in NLP : youtu.be/z5upsjfVU9c

#attention #nlp #transformers #selfattention #dotproduct #ai #datascience #datasciencegarage #softmax
@DataScienceGarage

ENCODER-DECODER Attention in NLP | How does it works - Explained

Data Science Garage 2022-09-30 | Attention in Natural Language Processing (NLP) is super important mechanism which makes your #AI systems such as chat bots, text predictions, sentimental analysis and more smarter and clever.
Good example is BERT. Attention is all you need! - as one famous article declares.

There are at least 4 different #Attention types in #NLP:
1. Encoder-Decoder attention (applied in Recurrent Neural Network - RNN).
2. Self attention.
3. Bi-Directional attention.
4. Multi-Head attention.

This video mostly explains the theory behind the encoder-decoder attention mechanism works. Here we will cover such aspects as:
- What is Dot-product in this NLP technique.
- Which role is taken by Similarity function for word to vector (wor2vec) approach.
- What is Alignment in attention calculation and how to calculate it.
- How to calculate a Dot-product (two different ways)
- Experiment with Python with word vectors (np.dot vs. np.matmul functions).

Finally, we will dive deeper into Dot-Product Attention with the schema which provides you intuition how this works. We will open this "black box"!

The content of the video:
0:00 - Intro
1:24 - Alignment with Dot-Product
5:04 - Experiments with Word Vectors in Python
12:06 - Dot-Product Attention (mathematical aspect)

In the last part of the video, we will get known about logical steps to calculate a Dot-Product for attention. In mathematical documentation it is called - matrix Z. This is our Attention product!
Here you will find such math operations such as matrix multiplication and transposing matrixes. Also you will find what roles takes embedding, linear, and softmax layers in the attention schema. Additionally, you will understand why we need to pay attention to PAD (padding) tokens as the input to encoder or decoder.

Read more:
1. Attention is all you need: official article in Arxiv: arxiv.org/abs/1706.03762
2. Attention and #Transformers: towardsdatascience.com/all-you-need-to-know-about-attention-and-transformers-in-depth-understanding-part-1-552f0b41d021
3. Attention Masks: lukesalamone.github.io/posts/what-are-attention-masks

Brilliant Course on Udemy, from which I learn all these things I showing you in this video: NLP With Transformers in Python by James Briggs (udemy.com/course/nlp-with-transformers)

NEXT LESSON: Self-Attention in NLP : youtu.be/pfvYuX0Gjys

#attention #dotproduct #naturallanguageprocessing #encoder #decoder #ai #artificialintelligence #machinelearning #datascience

@DataScienceGarage .

Github Action to AWS ECR (Docker Image) | Full Hands-on Tutorial

Data Science Garage 2022-09-09 | Data Science Garage presents a hands-on step by step tutorial: how to push a #Docker image which was built from Machine Learning (#ML) application directly in Github repository to AWS Elastic Container Registry (#AWS ECR). The provided steps are valid for any application stored in Github repository, for example for NodeJS, C++, React -based applications. The app. project must contains a Dockerfile which defines an instruction how to build a Docker image from that app.

With this tutorial you will learn how to write an YML (sometimes is called YAML) script to define your workflow. Github Action triggers this workflow file on push events in your Github repository. This approach is broadly used in Machine Learning Operations (MLOps) domain where ML engineers/ML developers builds solutions to automate application testing, deploying, monitoring and debugging. For DevOps this technique is also beneficial in designing CI/CD (Continuous Integration and Continuous Delivery) pipelines.

The parts of this video are:
0:00 - Solution scheme and theory
2:10 - Main facts about the application
3:56 - Start Setup Github action
6:31 - Create AWS User
8:01 - Handle AWS User credentials
9:39 - Create AWS ECR Repository
11:01 - Save Github Action and test workflow
12:03 - Github action test (2) - after changes made on the code
13:32 - Result: AWS ECR Images

Amazon Elastic Container Registry (Amazon ECR) is an AWS managed container image registry service that is secure, scalable, and reliable. Amazon ECR supports private repositories with resource-based permissions using AWS IAM. AWS ECR has many integration points with other AWS services, such as AWS SageMaker, AWS K8S. Additionally, you can bring your application images to AWS ECR by using MLflow API.
Official AWS ECR website: docs.aws.amazon.com/AmazonECR/latest/userguide/what-is-ecr.html

You can clone the repository used in this tutorial and replicate all the steps by yourself. Get the repository from here: github.com/vb100/github-action-to-ecr
Workflow file (main.yml) which declares steps of CI/CD pipeline: raw.githubusercontent.com/vb100/github-action-to-ecr/main/.github/workflows/main.yml

Once you have built your Docker images on AWS ECR, you can bring your application to production in several scenarios. You can combine it with Kubernetes, or if it is a ML-based application, you can deploy it in production by enabling batch predictions (I have prepared a video tutorial for that here: youtube.com/watch?v=FsoSBsrcx9Q)

If you have any comments or suggestions for the next tutorial/video, drop a comment below.
Thank you!

Cython vs. Python - 3k times faster code! | Test

Data Science Garage 2022-08-16 | With this tutorial we will compare the performance betwen Cython and Python. Theoretically, Cython is much faster than Python. But how much? How we can to compare these two trending technologies? This video demonstrates how to write a simple function in Cython, and generate an executable Python file from it.

Cython is an intermediate step between Python and C/C++. It allows you to write pure Python code with some minor modifications, which is then translated directly into C code. In other words, The #Cython language is a superset (the extension of for Python) of the Python language that additionally supports calling C functions and declaring C types on variables and class attributes. This allows the compiler to generate very efficient C code from Cython code.

The chapters of the video:
0:00 - What is Cython?
0:50 - Write a Python function to test
1:46 - Write a Cython function to test
3:12 - Generate executable shared Cython object (setup.py)
5:41 - Test Cython function with a script
7:06 - Compare Python and Cython functions execution speed

All what we will use is standard Python, terminal and file explorer (Finder on Mac/Windows explorer on Windows).
Do not forget to install Cython on your machine with terminal command: pip install Cython

Some theory the video touched:
- While there is def in Python to write a function, then in Cython we are having #cpdef. This is called - FUNCTION DECLARATION topic. Learn more about this here: notes-on-cython.readthedocs.io/en/latest/function_declarations.html
- Official Cython documentation: notes-on-cython.readthedocs.io/en/latest/index.html
- Python #Setuptools module: pypi.org/project/setuptools
- Cython installation: pypi.org/project/Cython
- timeit module in Python: docs.python.org/3/library/timeit.html
- Cython website: cython.org

To complete this tutorial, you can clone the video code repository from Github: github.com/vb100/cython_vs_python

All the best and good luck with Cython & Python!
Subscribe the channel to get more fresh high-quality videos first! Thanks!
Your -Data Science Garage.

Data Versioning Control with Real ML Project | Hands-On Lesson #1

Data Science Garage 2022-07-31 | This Hands-on tutorial demonstrated how to use Data Versioning Control (DVC) commands for real Machine Learning (ML) project. DVC is an important part of #mlops (Machine Learning Operations). You will learn how to initialize a new DVC session, how to prepare your data for tracking with DCV, how to read .dvc files, to understand what the information is stored in these files, and more. Also, once you watched this tutorial, you will know how to pull data from remote storage to your #datascience project.

Overall, this video shows the best practise how to work with DVC workflows for beginners and advances users (data scientists, data analytics, MLOps engineers).

DOWNLOAD THE FILES TO START THE TUTORIAL:
- You can fully follow the explained steps by yourself by cloning this Github repository to your local: github.com/vb100/dvc_project
- Training and Validation data: s3.amazonaws.com/fast-ai-imageclas/imagenette2-160.tgz

To complete this lesson, you will create a new branch on your Github repository where all data versioning control actions will be made.

You should understand that while combining Git and DVC, small files goes to Git, and large files goes to DVC control. Each control has it's own components, such as Git staging area, DVC cache, DVC remote and more.

The remote storage can be on the same computer (tutorial use-case) you are working on, or it can be in the cloud:
- AWS S3 Bucket.
- Google Cloud Bucket.
- Azure Blob storage, etc.

The content of the tutorial:
0:00 - Intro
1:17 - P1. Set-up your Python Environment
3:50 - P2. Hands-On the Basics DVC Workflow
6:50 - Tracking data files with DVC
9:27 - Uploading files to remote storage and push to DVC.
11:59 - Real life situation: Retrieve data from remote

Importants moments:
4:44 - Create a remote storage folder (dvc_remote) and connect it to DVC system for the data science project.
6:00 - Check config file in .dvc folder.
7:37 - What are .dvc files? (Explanation).
8:00 - What is MD5 decryption in DVC (Explanation).
9:08 - Git Control vs. DVC Control (Schemes).
10:55 - Check remote storage folder.
11:36 - check .dvc folder and config file in Github repository.
12:20 - Use dvc checkout command to pull data from remote storage.

Official DVC documentation: dvc.org

Thank you for watching!
Subscribe the channel to get more fresh similar content in future! See you there!

#github

Correlation Test with p-value? Yes! | Pandas vs. Scipy

Data Science Garage 2022-07-11 | Let’s talk about #Correlation test in Data Science with Python, Pandas and Scipy libraries. This video provides the overview between pandas.dataframe.corr() method and Scipy for calculating a coefficient of correlation for you features stored in Pandas DataFrame. The advantage comes from Scipy is that you will be able to evaluate the importance of the calculated correlation and decide which one is meaningful, and which ones are not for you business problem.

First of all, let’s be familiar with the official pd.DataFrame.corr() method documentation here: pandas.pydata.org/docs/reference/api/pandas.DataFrame.corr.html
As you can see, this way by using Pandas provides limited flexibility to use correlation for you Data Science project. There you can change the method of calculation (pearson, kendall, and spearman) and min_periods parameters only. So, it has minimal configuration.

From my personal experience, I suggest to try another way to play with correlation. You can test them with #pvalue which can be automatically calculated in you #Pandas Dataframe.

A mandatory warning that must be mentioned when talking about correlation is “Correlation does not imply causation”. You can read more about this warning in this Medium.com article: sundaskhalid.medium.com/correlation-vs-causation-in-data-science-66b6cfa702f0

You would often use correlation during exploratory data analysis (EDA) both for supervised and unsupervised ML problems to solve. The most straighforward way to calculate correlation is to use built-in Pandas method for that: pandas.dataframe.corr(). You can see the example in the video and in this Github code gist: gist.githubusercontent.com/KarunThankachan/f7b297630436042ca3e4f4c45f799b0a/raw/4f0bf26d10cc9d7ebafbc33abea09a69aaa30591/medium_corr.py

THE CORE OF THIS VIDEO:
When we talk about the correlation between variables, what we ideally want to measure is the correlation between variables in the entire population. However most data scientists work with a sample of data. So if we obtain a different sample it’s possible we could have different correlation scores. As such we need to assess the significance of the correlation values we calculated, which depends on the sample size.

By using Pandas approach, you will never get a p-values for your correlation, so you will be not able to test them. I suggest use Scipy library for this reason. Scipy’s stats library offers all three versions of the correlation test offered in pandas.DataFrame.corr():
- Pearson (docs.scipy.org/doc/scipy/reference/generated/scipy.stats.pearsonr.html)
- Spearman (docs.scipy.org/doc/scipy/reference/generated/scipy.stats.spearmanr.html)
- KendalTau (docs.scipy.org/doc/scipy/reference/generated/scipy.stats.kendalltau.html)
By using Scipy approach, you can pass columns of dataframe you want to compare. The Github snippet is here: gist.githubusercontent.com/KarunThankachan/6782cfedf3991248ca80c93332ca3d3f/raw/4fe0ec63a9d2598941b5e5c39ddbec9a5e3c17f5/medium_corr_v2.py

From the output of the code linked above, we can see the p-values and hence know how significant the correlations are.

Conclusion:
For correlation tests on data samples (which is often what you will be working on) always calculate the p-values as well. As such, when working with sample data go for the scipy.stats over pandas.DataFrame.corr()

Read more about:
- Pearson and Spearman correlation coefficients (clearly explained): towardsdatascience.com/clearly-explained-pearson-v-s-spearman-correlation-coefficient-ada2f473b8
- Kendall Rank Correlation (explained): towardsdatascience.com/kendall-rank-correlation-explained-dee01d99c535
- Interpreting correlations: towardsdatascience.com/eveything-you-need-to-know-about-interpreting-correlations-2c485841c0b8

Make Better Heatmap With Seaborn in Python

Data Science Garage 2022-06-19 | Let’s make heatmaps of correlation matrix with Python and Seaborn better. Half of a heatmap ıs useless. Anythıng dıstractıng should be oblıterated from a plot. Duplıcate cells of a heatmap are certaınly such a case as they appear to hold valuable ınformatıon.

This video demonstrates how to get rid of that clutter. To make this demonstration as simple as possible, I used two Jupyter notebooks. One left side I am showing a standard way how to build a Seaborn heatmap in a standard way.

One disadvantage on the left example that the generated heatmap represents duplicated values of calculated correlation coefficients in the perspective of diagonal line. You will see the view of this in the intro of this video.

On the right side, there is the improved technique how to visualize a heatmap of correlation matrixes. But without duplicated values we had on the left examples! For this improvement we used two additional lines of Python codes which we used to set-up a mask needed for this improvement.
1. numpy.triu_indices_from: Return the indices for the upper-triangle of arr. (numpy.org/doc/stable/reference/generated/numpy.triu_indices_from.html)
2. numpy.zeros_like: Return an array of zeros with the same shape and type as a given array (numpy.org/doc/stable/reference/generated/numpy.zeros_like.html).

Example Notebooks used in this video: github.com/vb100/corr_matrix_sns_experiment

The official Seaborn documentation: seaborn.pydata.org/tutorial/aesthetics.html
Generate correlation matrix with Pandas - pd.corr(): pandas.pydata.org/docs/reference/api/pandas.DataFrame.corr.html

I hope this way to display heatmaps in Python codes will help for data scientists and data analytics in their daily work with real world project!

#heatmap #seaborn #correlationmatrix

How to upload Pandas DataFrame to GCP Bucket with Python script

Data Science Garage 2022-06-05 | This video demonstrates how to upload Pandas DataFrame to Google Cloud Console (GCP) storage bucket. For this purpose we will use a very simple Python script, which uses the following dependencies:
- Pandas: pip install pandas(pandas.pydata.org/docs)
- Google Cloud Storage: pip install google-cloud-storage (pypi.org/project/google-cloud-storage)

Also, we will need to create a Service Account dedicated to communicate between our Python script and GCP bucket. For this Server Account me must to apply Storage Admin privileges. Furthermore, you need to create a new JSON based key which will be specified in your Python script. Use IAM dashboard in GCP to do it.

The Python script itself defines the main logic how our Pandas DataFrame is setup before uploading to the GCP bucker. There you must to specify:
- The the JSON key you generated in GCP IAM dashboard.
- Bucket name (created earlier).
- Source data file (in this example we are using CSV data file, where our Pandas DataFrame is saved on).
- Output directory and filename.

Python script template we used in the tutorial: github.com/vb100/pandas-to-gcp-bucket/blob/main/send_data_to_bucket.py

The content of the tutorial:
0:00 - Intro
0:27 - Create a new Google Cloud Bucket
1:05 - Create a new Service Account
2:13 - Create new JSON key for Service Account
2:40 - Create a new Python Virtual Environment
3:18 - Install dependencies on Virtual Environment
4:19 - Create a simple Pandas DataFrame
5:06 - Set-up a Python script for GCP
7:17 - Fix the Bucket name error
7:45 - Test the Python script
8:20 - Outro

See you on the next video!

#python #googlecloud #pandas

How to build ML Architecture with AWS SageMaker + Lambda + API Gateway | HANDS-ON TUTORIAL

Data Science Garage 2022-05-20 | This tutorial shows how to set-up a Machine Learning Architecture in Amazon Web Services by using combination of Amazon SageMaker, AWS Lambda Function, and Amazon API Gateway.

The main idea is to build a workflow in cloud where a regular end user from public internet can make requests on pre-trained ML model and get a prediction (or AI-based insights) in a second back.

From technical perspective, the backed works in such way: the the end use input his/her data and send it to API Gateway which uses REST API. Once the API service receive this input data, it forward it to pre-defined AWS Lambda function. Here, AWS Lambda function triggers the ML model endpoint which was created after the model training was completed and it's artifact saved in S3 bucket.
Finally, the ML endpoint after received user input data, make a prediction using a pre-trained ML model and send back a predicted value in the same way through AWS Lambda to Amazon API Gateway.

So, this video provides one of many ways how to deploy a Machine Learning model on the cloud using AWS.

In this video, the usage of Amazon API Gateway is demonstrated either by using AWS Console and a custom Python script (where you can flexible leverage the API functionality).

For demonstration purposes, in this video as an example I use breast cancer diagnosis prediction Machine Learning problem. The ML model endpoint was created specifically on this use case. You can find this notebook directly from AWS SageMaker Notebook templates.

Documentation used in this tutorial, where you can find snippets for IAM role and AWS Lambda code in Python programming language used in this video:
aws.amazon.com/blogs/machine-learning/call-an-amazon-sagemaker-model-endpoint-using-amazon-api-gateway-and-aws-lambda

Read more related to this use case:
- Amazon IAM: aws.amazon.com/iam
- Amazon SageMaker: aws.amazon.com/pm/sagemaker
- AWS Lambda Function: aws.amazon.com/lambda
- Amazon API Gateway: aws.amazon.com/api-gateway

- Also, I highly recommend to refresh you knowledge on IAM (Identity & Access Management), AWS Roles, Policies and Groups with this video: youtu.be/Ey9waqflzI4

The content of the tutorial:
0:00 - Explanation of ML architecture on AWS
2:15 - AWS Hand-on! Create AWS SageMaker Notebook Instance
4:49 - Explanation of Jupyter Notebook and train the ML model
8:17 - Build the SageMaker Endpoint for ML model
11:11 - Set-up AWS Lambda function
16:32 - Set-up Amazon API Gateway
19:05 - Test the ML architecture on AWS

I hope this tutorial was useful for you. If need any help, or wanna to share your experience or suggest a topic for the next video, please drop a comment below! - Thank you!

#sagemaker #lambdafunction #apigateway

Slava Ukraina!

Improve Your Python Code with dict.get() Method

Data Science Garage 2022-05-13 | Sometimes in Python instead using a classical if-else statement (which surely works well also), it is a good idea use some built-in Python methods and functions which makes your Python code even more professional. This #PythonTrick allows you manipulate with Python dictionaries even easily by keeping in mind that dictionaries is one of the most used data structures in Python.

Basically, dictionaries in Python are used to store data values in key:value pairs.
dict.get() is one of the mostly used method for dictionaries.

In this Python tutorial, I demonstrate how to improve your code with Dictionary Get functionality, which replaces if and else statement. In other words, you can replace multiple lines of codes with just one line of code which natively comes from Python programming language.

You can read additional resources on Python dict.get method in the following resources:
1. W3Schoolds: w3schools.com/python/ref_dictionary_get.asp
2. StackOverflow: stackoverflow.com/questions/11041405/why-dict-getkey-instead-of-dictkey
3. Official Python documentation: python-reference.readthedocs.io/en/latest/docs/dict/get.html

I hope this Python lesson will be useful either for advanced users, and beginners.

#datascience #python

How SHAP value is calculated? It is not hard! (simple example)

Data Science Garage 2022-04-25 | This video explains how to calculate a Shapley value with a very simple example. The Shap calculation based on three data features only to make this example as simple as possible. Also, you will be introduced to a main Shapley value formula, where we will calculate weights and marginal contributions.

The Shapley value is a solution concept used in game theory that involves fairly distributing both gains and costs to several actors working in coalition. Game theory is when two or more players or factors are involved in a strategy to achieve a desired outcome or payoff.

In many cases, the Shapley value can be replace a standard feature importance calculations provided by Scikit-learn, because the Shapley value can explain how the feature's values can impact the final prediction, and by how much they contributes to the Machine Learning predictions.

The video explains the Shapley value calculation by involving you to a simple example, where you analyze how Google Ads, Social Media, and Email marketing contributes. to prediction - if a user click on Ad, or not.

Shap value also ofter used in Explainable AI (XAI) to explain results to the business and understand the reasonings behind the ML models in terms of predictions and model performance.

The content of the video:
0:00 - What is a Shapley value and how to explain it?
2:14 - Simple Shapley value calculation example
3:00 - Understand Marginal Contributions for Shap
6:35 - Shap formula
7:13 - Calculate Marginal Contributions for Shap
11:20 - Calculate Weights for Marginal Contributions
13:31 - Calculate the final Shapley value
14:17 - Final summary of calculation results

Read more:
- Official Shap documentation: shap.readthedocs.io/en/latest/index.html
- Medium post: SHAP: Explain Any Machine Learning Model in Python (towardsdatascience.com/shap-explain-any-machine-learning-model-in-python-24207127cad7)

#shap #explainableAI #shapley

Get Started with Rust in Python with this Great Book! | Review

Data Science Garage 2022-04-15 | Improve and make you Python based codes with Rust. This book is a great resource to get started with Rust and make hands-on on you laptop easily. Many people thinks that learning Rust is hard, and ofter compare this programming language with C++ due its complexity. Maybe, but... Rust is a System language which can be leveraged with big efficiency if you facing issues of Multithreading, or creation of Python modules, or even using Github action for your Data science or DevOps projects. Overall, Rust is Memory Management Language (or a System language).

"Speed Up Your Python with Rust. Optimize Python performance by creating Python pip modules in Rust with PyQ3" is for those whole love challenges in coding and working with Python now.

The author of this book is Maxwell Flitton, who has been worked on numerous projects such as medical simulation software for the German government and supervising computational medicine students at Imperial College London. He also has experience in financial tech and Monolithic AI.

Rust can be installed directly from your terminal by using command: pip install RUST

The book consists of three big sections:
1. Getting to Understand Rust
- An introduction to Rust from a Python perspective
- Structuring code in Rust
- Understanding concurrency
2. Fusing Rust in Python
- Configuration setup tools for a Python pip module
- Packaging Python code in pip module
- Configuring continuous integration
- Creating a Rust interface for our pip Module
- Working with Python Objects in Rust
- Using Python modules with Rust
- Structuring an End-to-End Python Package in Rust
3. Infusing Rust into a Web Application
- Structuring a Python Flask App for Rust
- Injecting Rust into Python Flask app.
- Best practices for Integrating Rust

The content of the video review:
0:00 - Intro
0:51 - About the Author
1:09 - What this book is for?
1:29 - About me (data scientist)
1:46 - Few use cases from the book
2:19 - The structure of of sections
3:01 - What is Rust?
3:40 - Why Python with Rust?
4:51 - The structure of the book
6:04 - What else I like in this book
7:35 - Final word and conclusion

Additional references and resources:
- Book Github repository with real codes and examples to use: github.com/PacktPublishing/Speed-up-your-Python-with-Rust
- Official Rust repository on PyPi: pypi.org/project/RUST
- Comparative survey of the relative impact of mRNA features on local ribosome profiling read density: biorxiv.org/content/10.1101/018762v2
- The LinkedIn profile of the book author: linkedin.com/in/maxwell-flitton-9446a7119

BUY THIS BOOK HERE:
- packtpub.com/product/speed-up-your-python-with-rust/9781801811446
- amazon.com/Speed-Your-Python-Rust-performance-ebook/dp/B09LVL1T1Y

Thank you for watching!

#Rust #Multithreading #PythonModules

Deploy Flask application to AWS Elastic Beanstalk as a web app. | PART #2

Data Science Garage 2022-03-29 | This is the second part of developing machine learning (ML) project from zero to final deployment into the production. With this video we will deploy a pre-built Flask application to production environment with AWS Beanstalk.

AWS Elastic Beanstalk is an easy-to-use service for deploying and scaling web applications and services developed with Java, .NET, PHP, Node.js, Python, Ruby, Go, and Docker on familiar servers such as Apache, Nginx, Passenger, and IIS. Read more here: aws.amazon.com/elasticbeanstalk

There are many other solutions how you can deploy you ML model into the production, such as using Kubernetes with Docker, SageMaker, or use native solutions from other cloud providers (Google Cloud, Azure, or others). Why I choose AWS Beanstalk? - Because it is very fast, complete resource control, integration with Git and much more by side. However, it is not free, you would be billed with some invoices which covers your deployment infrastructure.

So, if you want to take a look how your Flask application will look like in production without doing many manual steps and time-consuming configuration, follow the video.

The content of the tutorial is:
0:00 - Step 1. Make a local application test
0:54 - Step 2. Specify app. dependencies
3:19 - Step 3. Build a Virtual environment
5:36 - Step 4. Adapt the application code
7:11 - Step 5. Add configuration file in .ebextensions
8:43 - Step 6. Zip your Flask application
9:26 - Step 7. Deploy application to AWS Beanstalk
11:38 - Test the final application in the cloud

Also, in this video you will learn:
- How to create a virtual environment (Mac)
- Find correct versions of dependencies with pip list command (2:25)
- Install dependencies from requirements.txt file (4:00)
- How to prepare a Flask code based on AWS Beanstalk requirements
- How to handle .DS_Store files in Mac laptops with ignoring them while zipping
- How to zip files with terminal command in Mac.
- How to specify a custom URL for your web application

Quick tips of the video:
- Each dependency can cover it's own deeper dependencies (you can find it with pip freeze command - 4:55)
- AWS Beanstalk requires "application" instead of "app"
- python.config configuration file is available here: github.com/vb100/KMeans-FlaskApp/blob/master/.ebextensions/python.config
- Python virtual environment is being created in AWS Beanstalk directly so we do not need to import it manually from a local machine.

The video tutorial of the Part #1: K-Means Python script to Flask application (youtu.be/HC7fdYyAcUc)

#beanstalk #flask #machinelearning

Thank you for watching!

Run Pandas as Fast as Spark - 10 useful practical functions

Data Science Garage 2022-03-18 | The aim of this video is to provide steps with the most used functions for managing DataFrames in Spark and their analogues in Pandas-on-Spark. Note that the only difference in syntax between Pandas-on-Spark and Pandas is just the import pyspark.pandas as ps line.

The Pandas API on Spark is a total game changer. Spark integrates a Pandas API so you can run Pandas on top of Spark.
we can obtain a wide range of benefits:
- If you use Pandas but you are not familiar with Spark, you can work with Spark right away, with no learning curve.
- You can have a single codebase for everything: small data and big data. A single machine and distributed machines.
- You can run your Pandas code faster.

Let's switch between Pandas, Pandas-on-Spark, and Spark.
The first thing we need to know is what exactly we are working with. When working with Pandas, we use the class pandas.core.frame.DataFrame. When working with the pandas API in Spark, we use the class pyspark.pandas.frame.DataFrame . Both are similar, but not the same. The main difference is that the former is in a single machine, whereas the latter is distributed.

This tutorial can be fully replicated on your side with standard Jupyter Notebook installed. Just follow the instructions from the begining.

So, with this video we will cover the following Pandas and Spark functions on dataframes (do not forget SQL, it is also needed a little bit!):
0:00 - Intro and importing PySpark libraries
2:57 - Import libraries and set a Spark session
4:28 - Load data and construct Pandas and Spark dataframes
6:48 - Function: SELECT
8:17 - Function: DROP DUPLICATES
9:41 - Function: FILTER
12:34 - Function: COUNT
14:28 - Function: DISTINCT
15:38 - Function: SORT
17:18 - Function: GROUP BY (for aggregation)
18:30 - Function: REPLACE
19:46 - Function: JOIN
20:36 - Outro and good bye!

Be notified, that before executing all steps explained in this tutorial, you must install PySpark module to your Python environment with pip install pyspark in your terminal.

Official PySpark documentation: spark.apache.org/docs/latest/api/python/getting_started/install.html

#pyspark #sql #dataframe

Install Jenkins on AWS EC2 Instance

Data Science Garage 2022-02-23 | This video shows how you can quickly install #Jenkins on AWS EC2 instance. Jenkins is an open source automation server which enables developers to build, test, and deploy their software. This tools also allows to setup and execute #MLOps (Machine Learning Operation) and #DevOps (Development Operation) workflows.

I think, it is important to learn how to set-up Jenkins on the instance, for example, on AWS EC2 Instance (Linux 2AMI - HVM). All steps with Linux commands are demonstrated there.

In oder to connect to our instance from the web browser, we will add two rules to Security Group:
- HTTP with port 80
- Custom TCP rule with port 8080 (to access instance directly from the web browser).

Content of the video:
0:00 - Setup EC2 Instance
1:59 - Connect to EC2 Instance from web browser
2:35 - Install Jenkins with Linux commands
4:19 - Access Jenkins with Password
5:34 - Open Jenkins interface and create Jenkins user

Used Linux commands (7 commands in total) to install Jenkins and other stuff on the instance:
1. Update yum for packages updating:
sudo yum update -y
2. Add Jenkins repository to install Jenkins from:
sudo wget -O /etc/yum.repos.d/jenkins.repo http://pkg.jenkins-ci.org/redhat/jenkins.repo
3. Import trust key from the Jenkins repository:
sudo rpm --import pkg.jenkins.io/redhat/jenkins.io.key
4. Install Java 1.8.0
sudo yum install java-1.8.0 -y
5. Install Extra Packages for Enterprise Linux (EPEL):
sudo amazon-linux-extras install epel
6. Install Jenkins tool on the instance:
sudo yum install jenkins -y
7. Start Jenkins tool:
sudo service jenkins start

To get a password to access Jenkins, I used this command:
sudo cat /var/lib/jenkins/secrets/initialAdminPassword

Jenkins allows you to trigger actions based on automatically detected code commits in a given repository. So, it is a cool tool to perform Continuous Integration and Continuous Delivery (CI/CD) pipelines for your MLOps project. If you need more tutorial on this, let me know by dropping a comment below. Thank you!

Send Slack Messages with This Python Script! - Clearly explained | Slack API Bot

Data Science Garage 2022-02-07 | With this video we will use Python and the SlackAPI to send automated messages to a Slack channel. For this we will build a Python based Slack Bot which uses Webhook URL for a selected channel and send customized messages based on the logic defined in your Python script.

I suggest to use few #Python functions which you can easily leverage for your business needs:
- The main() function provides a logic how the messages should be sent. You can customize this function upon your business requirements and extend the functionality as you want.
- send_slack_message() executes the message sending action to the specified Slack channel using the generated Webhook ID, which together with the message variable are sent to Slack API.

The tutorial demonstrated how to specify a payload which must be put into the POST request for the API. It is a key-values string formatted payload, where you should pay your attention to.

This Python tutorial does not requires any third-party dependencies, and uses the requests package only as the critical one.
Read more about request Python module here: docs.python-requests.org/en/master/user/quickstart

Also, I built this bot to be able to receive messages to be sent to Slack channel from a terminal. For this I used argument receiver functionality originally provided by standard Python package argparse.
Read more about argparse here: docs.python.org/3/library/argparse.html

This Bot was developed on Python 3.8, but I think it would work on other versions also.

Read more about Sending messages using Incoming Webhooks here: api.slack.com/messaging/webhooks

Table of the tutorial:
- 0:00 - Intro
- 0:23 - Build a Slack App and Get Webhook Token
- 3:32 - Coding part: build a Slack message sender with Python
- 8:57 - Set-up a Slack API payload variable
- 11:18 - Testing Python script, and debug development errors
- 13:25 - Sending more messages and customize the Bot logic

#SlackAPI #SlackBot

Kubernetes Explained | High Level Explanation

Data Science Garage 2022-01-31 | With this video I explain how #Kubernetes (k8s) works in high level with master and worker nodes, and more its components such as kubelet, kubectl, Service API and other.

Overall, Kubernetes is an open source system for automating the operations of containerized applications. Google created it an open-sourced in 2014.

With that, I explain why we use #Microservices, and Services inside our Kubernetes Cluster. Simple scheme also answers to the question, why we should want to start using Kubernetes?
The simple snippets of YAML (YML) files shows how a manifest (an instructions to the cluster) are written, and how we can align them to run our Kubernetes clusters properly.

Here I explain what is kubectl and kubelet. There are major components of Kubernetes you must to know and understand.

With this scope you will understand how we can setup our Kubernetes cluster (with Master and Slave nodes) to be accessed to the end user. For this you should understand the purposes of internal and external cluster IPs.

I found a very comprehensive series of videos dedicated to Docker and Kubernetes (provided by IBM Technologies), which helped me to prepare this video. Here I am sharing some of their videos below:
- Kubernetes vs. Docker: It's Not an Either/Or Question: youtu.be/2vMEQ5zs1ko
- Container Orchestration Explained: youtu.be/kBF6Bvth0zw
- Kubernetes Deployments: Get Started Fast: youtu.be/Sulw5ndbE88
- What are Microservices?: youtu.be/CdBtNQZH8a4
I hope it was useful for you.

If you liked it, please subscribe the channel to get more high quality content on datascience, #mlops and many related fields and domains!

AWS CodeCommit tutorial: your first Repo, Commit and Push

Data Science Garage 2022-01-19 | In this video I demonstrate how to create your first Git repository on AWS CodeCommit, and make your first commit and push to the repo. There are many similarities with Github, but also AWS CodeCommit introduces many advantages as well.

AWS CodeCommit is private Git repositories, featured by:
- Fully managed, highly available.
- Code only in AWS Cloud account (increased security and compliance).
- Secure (encrypted, access control, etc.).
- Integrated with Jenkins/AWS CodeBuild/ other Continuous Integration (CI) tools.
- Version control: is the ability to understand the various changes that happened to the code over time (and possibly roll-back), such as Git version control.
- Collaboration with other developers.
- Making sure that the code is backed-up somewhere.
- Making sure that it is fully viewable and auditable.
- No size limit on repositories (scale seamlessly).

AWS CodeCommit works in this way:
- The developers push the code often to their code repository and it can be expensive. The industry has several players that has Github (where Github offers a bunch of free public repositories), and also free one or paid private ones based on if you are a company or not, BitBucket. etc.

This tool can be integrated into CI/CD pipelines for your MLOps projects. In the context of AWS infrastructure, you can build these pipelines and workflows together with AWS Code Build, CodeDeploy, and CodePipeline with all benefits from these products.

Read more from official documentation:
- AWS CodeCommit: aws.amazon.com/codecommit
- AWS CodeDeploy: aws.amazon.com/codedeploy
- AWS CodeBuild: docs.aws.amazon.com/managedservices/latest/userguide/code-build.html
- AWS CodePipeline: aws.amazon.com/codepipeline

Knowing AWS CodeCommit is a one of the crucial topics in the preparation for the AWS DevOps Professional certification exam. More tutorials on this will be soon.

#aws #git #codecommit

Set-up a custom BERT Tokenizer for any language

Data Science Garage 2022-01-05 | BERT (Bidirectional Encoder Representations from Transformers) is a recent paper published by researchers at Google AI Language. BERT makes use of Transformer, an attention mechanism that learns contextual relations between words (or sub-words) in a text. In its vanilla form, Transformer includes two separate mechanisms — an encoder that reads the text input and a decoder that produces a prediction for the task.

In this video our tokenizer will be based on BERTWordPiece, which is the subword tokenization algorithm used for BERT, DistilBERT, and Electra. WordPiece first initializes the vocabulary to include every character present in the training data and progressively learns a given number of merge rules. In contrast to BPE, WordPiece does not choose the most frequent symbol pair, but the one that maximizes the likelihood of the training data once added to the vocabulary.

In many NLP (Natural Language Processing) cases we meet challenges with non-English text. In this BERT with Python tutorial, I used non-cleaned comments text data. This text data fully is in Lithuanian, not in English. This is where the challenge is coming on. We need to create a custom vocabulary (vocab.txt) file for a BERT tokenizer to train a pre-trained BERT model to solve our NLP problem. These processes can be split into four separate steps, as follows:

0:00 - Intro
0:37 - Step 1. Set-up a Python virtual environment
2:17 - Step 2. Prepare text data for training
6:11 - Step 3. Train a BERT tokenizer
9:25 - Step 4. Use a BERT tokenizer

These steps can be easily adopted to your unique NLP cases, or to develop a new BERT tokenizer by taking some crusial moments from this tutorial. By following this tutorial you should understand basically how BERT works, also you should be able to play with Python scripts to develop a new ones codes for you, and finally you should be familiar what is special tokens in BERT. At the end of this this video I returned values as PyTorch tensor, you can play around it and make it differently if you like to do it. At the end we will save a BERT model locally just to show how it is works for you. Links and resources mentioned in the video are listed below:

- BERT Tokenizer official documentation: huggingface.co/docs/tokenizers/python/latest/quicktour.html
- Tokenizer train method usage: huggingface.co/docs/tokenizers/python/latest/api/reference.html#tokenizers.trainers.Trainer
- BERT special tokens, good article to read more: analyticsvidhya.com/blog/2021/05/all-you-need-to-know-about-bert
- BERT base uncased model for download: huggingface.co/bert-base-uncased/tree/main
- BERT tokenizers: huggingface.co/docs/transformers/main_classes/tokenizer
- BERT from_pretrained method: huggingface.co/docs/transformers/v4.15.0/en/main_classes/model#transformers.PreTrainedModel.from_pretrained
- BERT paper: arxiv.org/pdf/1810.04805.pdf

By completing this tutorial you will be able to develop a unique BERT tokenizer and adopt it for your NLP task. Also you can use your customer tokenizer in text classification (or multi-classification) with BERT. The complete code is clear to understand even for beginners of BERT like me, so I hope that the video will be easy to follow for all watchers.

#BERT #NLP #Python

Dockerize Flask API application with Dockerfile and Docker Compose

Data Science Garage 2021-12-20 | This video explain how to:
- Test your Python API application to make predictions with ML model endpoint with Postman.
- Create and set-up Dockerfile (with clear explanation).
- Prepare Docker compose (docker-compose.yml) file to orchestrate services need to run your Flask application (for example, a database).
- Build and run a Docker image from Docker compose file.

You will also learn what are the differences between docker compose build and docker compose up commands. With that you will get a clear explanation how Dockerfile works, and what is the logic behind. So, you will quickly dive into Dockerhub - the big library of Docker images which can be used for your unique Docker containers. This is what exactly we are doing in this tutorial.

By using this approach you will be able to Dockerize any Flask based API app, whether it is ML classifier/model or a simple API. In this example I used a predefined model.joblib ML classifier previously.

The content of the video:
0:00 - Test Flask API application locally with Postman
2:57 - Explain Dockerizing and setup Dockerfile
9:57 - Setup Docker Compose file
13:54 - How to setup Docker Compose for multiple Services
15:25 - Build and Run a Docker Image

If you wish to get more videos about MLOps, Docker for Machine learning, or something for related fields, subscribe and let me know in the comment section below!

#docker #flask #dockerfile

NeRF - Neural Radiance Fields for View Synthesis (2D to 3D) - DEMO

Data Science Garage 2021-12-16 | NeRF stands for Neural Radiance Fields for View Synthesis. This problem can be set to the Computer Vision (CV) domain and now the demand of NeRF is growing.

In this video I demonstrate the quick results of minimal implementation of volumetric rendering provided at keras.io/examples/vision/nerf (the code is written in Keras with Python).

Example code is here: keras.io/examples/vision/nerf

The NeRF is a method which achieves state-of-the-art results for synthesizing novel views of complex scenes by optimizing an underlying continuous volumetric scene function using a sparse set of input views.

Input to neural network:
- spatial location (x, y, z).
- viewing direction (teta, gama).

In NeRF there are used fully-connected neural network which outputs:
- output colos.
- output density.

A scene is a fully-connected (non-convolutional) deep network. This helps to represent a scene as a continuous 5D function.

NeRF is also promising in fields such as AI for Medicine, Geometry Visualization, View-Dependent Appearance, creating 3D meshes, capturing 360 scenes, Positional Encoding and much more.

Good to read more:
- Original paper: NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis (arxiv.org/pdf/2003.08934.pdf)
- One of the authors's super page about NeRF: matthewtancik.com/nerf

Compare ML Models in few clicks with PyCaret in Python - DEMO

Data Science Garage 2021-12-10 | PyCaret is an open source, low-code machine learning library in Python that allows you to go from preparing your data to deploying your model within minutes in your choice of notebook environment.

In other words, PyCaret is an open-source, low-code machine learning library in Python that automates machine learning workflows. It is an end-to-end machine learning and model management tool that speeds up the experiment cycle exponentially and makes you more productive.

PyCaret is an alternate low-code library that can be used to replace hundreds of lines of code with few lines only.This Python package supports following Machine Learning (ML) models and frameworks:
- scikit-learn
- XGBoost
- LightGBM
- CatBoost
- spaCy
- Optuna
- Hyperopt
- Ray, and few more.

In this Demo I showed how to:
- Install PyCaret.
- Setup PyCaret environment in Jupyter Notebook
- Prepare data for analysis.
- Compare different ML models for the given dataset
- Analyze ML performance metrics
- Select the best ML model
- Check the model predictions
- Save the best model locally on your computer.

Links:
- Official PyCaret documentation: github.com/pycaret/pycaret
- Official PyCare Github repo: github.com/pycaret/pycaret

At the moment this video was recorded, there was PyCaret v.2.3 available. The project is updated very smartly and more interactive features will be available soon. Recommend to try it!

How to install Kubernetes on Ubuntu 18.04 | 9 STEPS | explained

Data Science Garage 2021-11-23 | This tutorial explains how to set up your Kubernetes cluster on Ubuntu 18.04 server. For this example I use AWS EC2 (Cloud Computing Service from Amazon Web Service), but the provided steps are applicable for any Ubuntu 18.04 environment. This tutorial begins on how to prepare your EC2 instances, then run it all and connect to all these from your local machine - in my case, I used MacOS. If you are Windows user, for this step you can use PuTTy service.

For this example, I will create 1 Master node and 2 Worker nodes (in total 3 instances, or 3 nodes) for my Kubernetes Cluster. Also, I am sharing all related links to resources which helped and were useful in preparing this tutorial. There are many tutorials and instructions how to set up Kubernetes, but I tried to prepare the best one! So, in total - 33 commands and 9 steps for Kubernetes installations, everything is for you.

The plan of attack / Steps to Set up your Kubernetes cluster are:
0:00 - Intro
0:10 - Step 1. Set Up AWS EC2 Ubuntu 18.04 Instance

3:33 - Step 2. Connect to EC2 Instances from local terminals
4:34 - 2.1. Make your private key (.pem) executable
5:18 - 2.2. Connect to all nodes from instances

6:44 - Step 3. Install dependencies and configure Kubernetes
7:01 - 3.1. Update the package list
7:39 - 3.2. Install Docker
8:56 - 3.3. Check Docker version
9:32 - 3.4. Start and enable Docker

11:10 - Step 4. Install Kubernets on Ubuntu 18.04
11:20 - 4.1. Add a Signing Key (GPG)
12:24 - 4.2. Add the Kubernetes software repository

13:14 - Step 5. Prepare Kubernetes installation tools
13:35 - 5.1. Install kubeadm, kubelet, and kubectl
14:57 - 5.2. Check if kubeadm, kubelet, and kubectl are on hold
15:49 - 5.3. Check kubeadm version

16:30 - Step 6. Kubernetes Deployment
16:40 - 6.1. Disable the swap memory from nodes
17:03 - 6.2. Set host names for each node

18:20 - Step 7. Workaround to avoid mismatches between Docker drivers
19:04 - 7.1. Create a daemon.json file
22:20 - 7.2. Reset Docker and kubeadm services

23:44 - Step 8. Initialize Kubernetes Pod Network
24:16 - 8.1. Set up a Pod Network for Master node
26:44 - 8.2. Create a directory for the initialized Master node
27:05 - NETWORK ISSUES - you can skip it
28:16 - 8.3.Set up a virtual network for communication across nodes (flannel)
29:04 - 8.4. Check the pods status
29:30 - 8.5. Join Worker nodes to a Master node

30:34 - Step 9. Check final stages
30:55 - 9.1. Confirm that Master node communicates with all Worker nodes
31:47 - BONUS: Github repository - Readme file with all steps in one place

- - - - - - - - - -
Additional resources:
- You can find all steps provided in this video clearly explained in Readme right here: Github repo with all steps: github.com/vb100/Deploy_Kubernetes_AWS_EC2/blob/main/README.md
- How to set up AWS EC2 and run Python scripts (step by step example): youtu.be/qUHQuZjTOFA
- Download PuTTy for Windows: putty.org
- More about workaround step (7): github.com/kubernetes/kubernetes/issues/75803
- Kubernetes networking model: ibm.com/docs/en/cloud-private/3.1.2?topic=networking-kubernetes-network-model
- What is a Pod in Kubernetes: w3schools.com/html/html_images.asp

I hope that this tutorial was not too long (if yes, let me know, I will do shorter in future). Also, any feedbacks and comments are welcome!

Vytautas - Data Science Garage.

#kubernetes #docker #devops

Python for Geeks. Book review with comments (2021)

Data Science Garage 2021-10-31 | New Python book is here! Python for Geeks. Build production-ready applications using advanced Python concepts and industry best practices. This book is written by Muhammad Asif (you can reach him here: linkedin.com/in/muhammad-asif-phd-2012).

With this book, you can learn:
- Understand how to design and manage complex Python projects.
- Strategize test-driven development (TDD) in Python.
- Explore multithreading and multiprogramming in Python.
- Use Python for data processing with Apache Spark and Google Cloud Platform (GCP).
- Deploy serverless programs on public clouds such as GCP.
- Use Python to build web applications and applications programming interfaces.
- Apply Python for network automation and serverless functions.
- Get to grips with Python for data analysis and machine learning.

This book is released in August 2021 and provides almost 500 pages of fresh content, including 14 chapters and 4 sections.
1. Python, beyond the Basics.
2. Advanced Programming Concepts.
3. Scaling beyond a Single Thread.
4. Using Python for Web, Cloud, and Network use cases.

I found that the author explains such topics which are very specific and rarely mentioned in other programming books. I mean topics such as Polymorphism in OOP, Multiprocessing, Asynchronous Programming, Operating on multiple files and much more.

I think this book is a good investment for all kind of Python users: beginners in Python, ML engineerings, DevOps specialists, Advanced users, Python developers and all, who are interesting in coding with Python.

This book is published by Packt and available in Amazon to buy: amazon.com/Python-Geeks-production-ready-applications-practices/dp/1801070113

#pythonbook #pythonapplications #developing

Hyperparameter tuning for Ensemble of ML models (Simple Python Example)

Data Science Garage 2021-10-26 | If you have created an ensemble of ML models in scikit-learn, and you want to improve its performance even further, you can tune this ensemble itself.It can be easily implemented by GridSearchCV and VotingClassifier from Scikit-Learn. The main idea is to set the hyperparameters you want to search the best combination of values on, then set all possible weights of Machine Learning classifiers you have in your situation. For example, one option could be that your Random Forest classifier is more important that Linear Regression classifier, and opposite.Then the search for the best combination of your selected hyperparameters will do almost all the job.

That means that by tuning ensembles of ML models you can get the same accuracy as you got from cross validation score in ensemble learning mannier, or increase it with the tuning procedure. So, the good idea is to always invest some time tuning your ensemble.

In this example we used following ML models for binary classification problem:
- Logistic Regression classifier: scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html
- Random Forest classifier: scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html
- Decision Tree classifier: scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html

Also we used following methods originally from scikit-learn:
- GridSearchCV: scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html
- Voting Classifier: scikit-learn.org/stable/modules/generated/sklearn.ensemble.VotingClassifier.html
- Cross Validation Score: scikit-learn.org/stable/modules/generated/sklearn.model_selection.cross_val_score.html

By using all of these functionalities we implemented the search across Soft and Hard voters in Grid Search CV. The main differences and examples of calculations are presented in the BONUS part of this video.

The content of this video:
0:00 - Creating Ensemble of Machine Learning models1:00 - Tuning the Ensemble of ML models
9:58 - Results of tuning (accuracy).
10:56 - BONUS part: Soft Voting and Hard Voting

#hyperparametertuning #ensemblelearning #gridsearchcv

Enjoy!

How to Use apply() function to Return Multiple Columns in Pandas DataFrame

Data Science Garage 2021-10-21 | This tutorial demonstrated how to generate separate multiple Pandas DataFrame columns from single column. For example, if you have name and surname in one feature columns on Pandas DataFrame, you can automatically build two additional columns with name and surname separately.

For this we will use a simple Python function returning two values at the end. Then apply function is applied to our Dataframe incorporating zip statement with single-star key argument. This is only one line of Python line.

In our example, we just use a split function to simply split the given string argument in order to return two separate values out of our Python function.

This solution valid in any kind of #Pandas #Dataframe and can easily be applied in #datacleaning, feature engineering, or data exploration steps in your real data science or data analytics tasks.

Useful links:
- Pandas apply() function: pandas.pydata.org/docs/reference/api/pandas.DataFrame.apply.html
- Python zip statement: w3schools.com/python/ref_func_zip.asp
- Python key argument of one-star: stackoverflow.com/questions/36901/what-does-double-star-asterisk-and-star-asterisk-do-for-parameters

Enjoy!