AWS Glue development using notebooks in SageMaker

Modern data engineering often involves building pipelines that are efficient, scalable, and easy to manage. AWS Glue is a serverless data integration service designed to make it simple to discover, prepare, and transform data for analytics and machine learning. While AWS Glue provides its own development interface, one powerful and flexible way to work with Glue jobs is through SageMaker Studio Notebooks.

In this blog, we’ll explore how to develop AWS Glue scripts using SageMaker notebooks, including setup, code samples, and best practices.


Why Use SageMaker Notebooks for AWS Glue Development?

While AWS Glue Studio offers a visual interface and script editor, SageMaker notebooks provide a more interactive, code-first development experience. This setup is perfect for data engineers and scientists who:

  • Want to explore datasets interactively
  • Prefer to use Jupyter-based notebooks
  • Need a flexible Python environment
  • Integrate Glue with machine learning pipelines

Prerequisites

  1. Before you begin, ensure you have the following:
  2. An AWS account with access to Glue, SageMaker, and S3
  3. A SageMaker Studio environment configured
  4. Proper IAM roles with permissions for Glue, S3, and SageMaker
  5. Also, you’ll need to enable Glue interactive sessions in your AWS account.


Step 1: Set Up Glue Interactive Sessions

AWS Glue interactive sessions allow you to run Glue jobs in real time, making it ideal for notebook development.

To enable:

Open AWS Glue Console.

Navigate to Notebooks > Interactive sessions.

Choose a role with permissions (or create one with AmazonS3FullAccess, AWSGlueServiceRole, and AmazonAthenaFullAccess).


Step 2: Launch SageMaker Notebook

Open Amazon SageMaker Studio.

Create a new Python 3 notebook.

Install the AWS Glue libraries using the following code:


python


!pip install --upgrade aws-glue-sessions boto3

Import necessary libraries:


python

Copy

Edit

import sys

import boto3

from awsglue.utils import getResolvedOptions

from awsglue.context import GlueContext

from pyspark.context import SparkContext

Note: Some setups may require you to launch a kernel with PySpark pre-installed.


Step 3: Connect to a Glue Session

You can start an interactive session directly from the notebook using:


python

Copy

Edit

from awsglue.context import GlueContext

from pyspark.context import SparkContext


sc = SparkContext.getOrCreate()

glueContext = GlueContext(sc)

spark = glueContext.spark_session

This allows you to write PySpark or Glue DynamicFrame code inside your notebook.


Step 4: Develop and Test ETL Code

Now you can write ETL logic interactively. Example: reading from an S3 bucket and applying transformations.


python

Copy

Edit

datasource = glueContext.create_dynamic_frame.from_options(

    connection_type="s3",

    connection_options={"paths": ["s3://your-bucket/input-data"]},

    format="csv",

    format_options={"withHeader": True}

)


# Transformation

transformed = datasource.drop_fields(["unnecessary_column"])


# Write back to S3

glueContext.write_dynamic_frame.from_options(

    frame=transformed,

    connection_type="s3",

    connection_options={"path": "s3://your-bucket/output-data"},

    format="parquet"

)

Step 5: Convert to Glue Job (Optional)

Once your notebook code is ready, you can convert it into a Glue job script:

Export the notebook as a .py file.

Upload the script to S3.

Create a Glue job in the console pointing to the script.


Best Practices

Use Glue bookmarks to track processed data.

Store configurations in parameterized cells for reusability.

Use logging to monitor job performance.

Leverage Git integration in SageMaker for version control.


Conclusion

Using SageMaker notebooks for AWS Glue development combines the best of both worlds: the power of Glue’s managed ETL service and the interactivity of Jupyter notebooks. This approach is ideal for data engineers and scientists building sophisticated, scalable data pipelines with full control and flexibility.

Learn AWS Data Engineer with Data Analytics
Read More: Merging delta records in Redshift using UPSERT


Visit Quality Thought Training Institute in Hyderabad
Get Direction

Comments

Post a Comment

Popular posts from this blog

Tosca vs Selenium: Which One to Choose?

Choosing the right test automation tool can make or break the efficiency of your QA process. Two widely discussed tools in the test automation space are Tosca by Tricentis and Selenium, the open-source web automation framework. While both tools aim to automate testing and accelerate software delivery, they cater to different needs and teams. In this blog, we’ll provide a comprehensive comparison between Tosca and Selenium to help you decide which one suits your project best. Overview Selenium is an open-source framework primarily used for automating web applications. It supports multiple languages (Java, Python, C#, etc.) and browsers, making it ideal for custom, flexible automation solutions. Tosca, on the other hand, is a commercial test automation tool that supports model-based testing, API testing, mobile, desktop, and even SAP testing. It's designed for enterprise-grade testing with minimal scripting. Key Comparison: Tosca vs Selenium Feature          ...
Read more

Flask API Optimization: Using Content Delivery Networks (CDNs)

When building APIs with Flask, performance and speed are key metrics that directly affect user experience and application scalability. One powerful but often overlooked way to enhance these metrics is through the use of Content Delivery Networks (CDNs). While CDNs are traditionally associated with delivering static assets like images, CSS, and JavaScript, they can also play a significant role in optimizing your Flask-based APIs. What Is a CDN? A Content Delivery Network is a globally distributed network of proxy servers that cache content closer to end-users. Instead of every user hitting your central Flask server, a CDN enables them to access cached responses or assets from a nearby server, significantly reducing latency and improving response times. Benefits of Using CDNs with Flask APIs 1. Reduced Latency Latency is the time it takes for a request to travel from the client to your server and back. By caching static or semi-static responses at edge locations, CDNs reduce the distance...
Read more

Using ID and Name Locators in Selenium Python

When automating browser tasks with Selenium in Python, one of the first challenges is identifying and interacting with web elements. Selenium offers multiple strategies for locating elements, and two of the simplest and most effective methods are using the ID and Name locators. These locators are fast, reliable, and easy to use, making them ideal for beginners and experienced testers alike. In this blog post, we’ll dive into how ID and Name locators work in Selenium Python, provide examples, and share best practices to help you write clean and maintainable test scripts. What Are Locators in Selenium? Locators in Selenium are methods used to find elements on a web page. Selenium supports several locator types, including: ID Name Class Name Tag Name Link Text CSS Selector XPath Among these, ID and Name locators are often the most efficient because browsers optimize them for speed and accuracy. Setting Up Selenium with Python Before using locators, ensure you have Selenium installed: bash...
Read more