Introduction

This article explains how to set up applications running on AWS for compute-heavy tasks that execute on-demand rather than continuously. While ad-hoc jobs don’t require constant deployment, they do need the newest code version available whenever they run—making continuous delivery essential.

Readers are directed to read part 1 for foundational setup information, including machine configuration and code deployment mechanisms using systemd.

The Data Script Scenario

The piece describes a practical use case: a NodeJS script running on a powerful EC2 instance that performs data operations. The script:

• Retrieves data from multiple online sources
• Reads and processes CSV files
• Compares data against an in-memory database
• Produces large datasets for pipeline consumption
• Could alternatively handle resource-intensive builds

The author notes that ideally these tasks would be completely code-agnostic, allowing dozens to run simultaneously with different code—but acknowledges this is a complex topic for another discussion.

Running the Task

Basic Approach:

Node scripts execute via node some-script.js, running to completion then exiting. The proposed architecture:

• Script runs inside a Docker container
• Docker operates on an EC2 instance booted by systemd as a daemon
• Instance automatically shuts down once the job completes

Setup Instructions

Code Example

The sample Node script demonstrates:

const winston = require('winston');
const aws = require('aws-sdk');
aws.config.update({region: 'your_region'});
winston.level = 'debug';
winston.log('info', 'Pushing to S3'); 
const s3 = new aws.S3();
var ec2 = new AWS.EC2();
const myBucket = 'bucket.name'; 
const myKey = 'myBucketKey';
const shutDownEC2 = (instance_id) => {
    winston.info("Shutting down");
    ec2.stopInstances({
        InstanceIds: [ instance_id ]
    }, function(err, data) {
            if (err) {
                winston.error(JSON.stringify(err))
                // Trigger some alerting here
            } else {
                winston.info('Done')
            }
       }
    );
}
s3.createBucket({Bucket: myBucket}, function(err, data) { 
    if (err) {  
        // Creating bucket failed
        winston.log(err);
        shutDownEC2('ec2_id');
   } else {    
       const params = {
                  Bucket: myBucket, 
                  Key: myKey, 
                  Body: 'Hello!'
               };      
        s3.putObject(params, function(err, data) {          
            if (err) {
                // Putting object failed
                winston.log(err);
                shutDownEC2('ec2_id');
            } else {   
                // It was successfully done
                winston.log("Successfully uploaded data to myBucket/myKey");
                 shutDownEC2('ec2_id');
            }       
        });    
    } 
});

Key Considerations:

• The instance ID represents the EC2 instance executing the code
• Shutdown must trigger on every process exit—both success and failure scenarios
• Implement comprehensive try/catch blocks with logging and alerts to prevent expensive instances remaining running after exceptions

Docker Configuration

FROM node:boron
WORKDIR /usr/src/app
COPY package.json .
RUN npm i
COPY . .
CMD ["node", "script.js"]

This Dockerfile optimizes caching by copying package.json and installing dependencies before adding remaining files.

Control Mechanisms

Once configured, the EC2 instance remains suspended until triggered. The system requires mechanisms to start it on-demand.

Starting the Job

AWS Lambda provides the ideal trigger mechanism using the AWS JavaScript API. The author recommends two tools:

• Claudia.js (https://claudiajs.com/) — fast, Lambda-focused, optimal for AWS-only work
• Serverless Framework (https://serverless.com/) — more comprehensive, multi-cloud capable, slower during development due to CloudFormation dependency

Lambda code to trigger EC2 startup:

'use strict';
// Load the AWS SDK for Node.js
const AWS = require('aws-sdk'); 
const ec2 = new AWS.EC2(); 
module.exports.handler = (event, context, callback) => {   
    const params = {
        InstanceIds: [ 'instance_id' ],
        AdditionalInfo: 'START IT UP'  
    };  
    ec2.startInstances(params, function(err, data) {
        if (err) {
          callback(err, null);
        } else {
          callback(null, {
              statusCode: 200,
              body: JSON.stringify({
                 message: 'STARTING'
              })
          })
         }
      });
};

Stopping the Job

Reuse the shutDownEC2 function in a Lambda function for manual shutdown or emergency stopping if tasks become stuck.

Task Scheduling

CloudWatch Events handles scheduling:

1. Navigate to CloudWatch → Events → Rules
2. Select scheduling option (interval or cron expression)
3. Add the Lambda function as a target
4. Configure and create the rule

Users can schedule tasks at desired intervals, from hourly to daily to custom cron patterns.

Practical Applications

The combination of these tools enables:

• Custom CI/CD systems using Lambda triggers and EC2 build machines
• Automated database backups on schedules
• Regular web crawling tasks

Conclusion

This collection of accessible, cost-effective cloud tools provides “powerful tooling to setup and maintain the most commonly needed dev-ops tasks,” making enterprise-grade automation achievable for various scenarios.