Difference between revisions of "Test Emails"

Latest revision as of 21:18, 13 November 2017

GWAVA's products are all about messages in email systems. Archiving them, backing them up, or analyzing them. This page is about generating emails for your test system. You will need to have some test users in place.

[edit] Download GoPostal

Creating test emails is tedious, download GoPostal. These are applications for Linux and Windows that automatically generate test emails.

GoPostal will create a randomized subject, body, and attachment as well as sending a random file from the dataset folder. You will have to add more files to the dataset folder.

Data for the dataset folder:

• Project Gutenberg
• NASA
• pdftextfiles
• Amazon

Use cron or Task Scheduler to do it on a recurring schedule so you don't have to worry about it.

The history and design of GoPostal is below.

[edit] Naming Test Users

Since there is a chance your test system may end up being used to create documentation it is advised to stay away from famous, offensive or copyrighted names.

There are online fake name generators [1] but for the sake of these scripts something simple like user0 or test0 is great since we can use concatenation to have any number of usernames, i.e. user0-user9.

Example names using the most popular gender neutral first names and most common last names from around the world:

• Aiden Li
• Blake Zhang
• Carter Wang
• Dakota Nguyen
• Eden Garcia
• Finley Gonzalez
• Hayden Hernandez
• Jayden Smith
• Kamryn Smirnov
• Riley Muller

[edit] Creating Test Emails

You will need messages for your email system and there are two basic ways of doing that:

1. Manually
2. Automatically

[edit] Manual Email Creation

Everyone knows how to do this.

1. Open the email client
2. Choose one or more recipients
3. Give it a subject: test
4. Bang on the keyboard a few times for a message body
5. Send it off.

That gets old really fast.

It is a little more fun and is handy for some tests is to use a lorem ipsum generator [2] to create body text and translate it into different alphabets [3] Like Arabic, Chinese, Hebrew, Guarati and Sindhi.

[edit] Automatic Email Creation

Getting a few emails into your system is nice but a continuous stream of emails is much more helpful. Also, Retain uses single-instance storage and so any identical message bodies or attachments are only saved once, so we need something that will provide us with lots of unique material.

Linux bash and Windows PowerShell are scripting languages that able to generate emails for you with a little forethought.

[edit] Linux

[edit] Download package

Download the gopostal.sh script package from [4] This has some more features than the script below, including multilingual html attachments, more interesting subject lines and logging.

Prerequisites: Go into YaST and enable the mailserver, because this depends on the mail command being available. The sending mailserver must be configured with: connection type Permanent, Outgoing Mail set to the IP address of your receiving mail server. To send to a different mailserver you will need to configure the DNS to with the proper A and MX records.

Required: You need to specify your domain and the users the script should send to. There is an array of usernames, replace them with users that exist in your system.

Optional: The number of messages to send each time this script is run can be changed to more than 1. 4-5 per hour per user will provide average amounts of mail an average user receives. The message body size for each message (1023 bytes by default) can be set as well. Logging can be enabled or not. The log file size is limited to 10MB so it doesn't fill the server.

Otherwise, you can use the script below for a simpler email.

[edit] Text Source Material

First, you are going to need a source text. Something large enough to provide plenty of material but isn't copyrighted. This is what I use [5] and Project Gutenberg [6] has plenty of public domain material to choose from.

Save it as *textdata.dat*.

[edit] Email Generator Script

This is the main script you will want to use. You can automate it with crontab -e to create a recurring task that runs the script.

#!/bin/bash
#gopostal.sh
# Stephan Fassmann 03/31/2016
# Sends a number of random emails to test users.

#This script is designed to produce and send a large amount of unique email for the testing of the database and storage area of Retain. 
#The body is shuffled large text file. The attachment is a shuffled csv file. This makes the data searchable and unique. 

#REQUIREMENT: Set your email domain!
#REQUIREMENT: Enable mail in YaST. The sending mailserver must be configured with: connection type Permanent, Outgoing Mail set to the IP address of your receiving mail server. 
#REQUIREMENT: The receiving mail server must have the target users (user0-9 by default). 
#IMPORTANT: This script has no error checking or other safeties in place. It expects you to know what you are doing. 
#WARNING: For internal use only. This will send lots of email, fast. This looks like a simple spambot and most real world mail servers are configured to block domains that spam.

#use crontab -e  [00 */1 * * * /dev/gopostal/gopostal.sh] to send hourly, set LOOP to 4 for 96 messages a day per user which is average.

#REQUIREMENT: Set your email domain!
DOMAIN=your.email.domain

#Set the location of the script and the support files
FILELOCATION=/dev/gopostal

#Specify the number of messages to send. By default will send 4 messages to each user each time the script is run.
LOOP=4

#Specify the usernames and numbers to send to. e.g. you can change it to test with any number of user numbers. default user0-user9.
USERNAME=user
USERNUM=9

#How large to fill the body of the message, in bytes, 1023 by default.
BODYSIZE=1023

#FILES
CODEFILE=$FILELOCATION/gopostal.sh
#Data files required for creating the body.
INFILE="$FILELOCATION/textdata.dat"
#OUTFILE and BODY are shuffled and truncated output files. These files are temporary.
OUTFILE="$FILELOCATION/shuffledtext.tmp"
BODY="$FILELOCATION/bodytext.tmp"

#Data files required for creating the csv attachment. You can replace with any large comma delimited files.
INDATA="$FILELOCATION/csvdata.dat"
HEADER="$FILELOCATION/csvheader.dat"
#SHUFDATA and OUTDATA are shuffled and truncated output files. These files are temporary.
SHUFDATA="$FILELOCATION/shufflecsv.tmp"
OUTDATA="$FILELOCATION/randcsv.tmp"
#sendfile.csv is the completed randomized csv attachment. This file is temporary.
ATTACHMENT="$FILELOCATION/sendfile.csv"

#Logging for developmental troubleshooting
#log: when did it start
LOGFILE=$FILELOCATION/gopostal.log
echo "$(date "+%m%d%Y %T"): Beginning"
echo "$(date "+%m%d%Y %T"): Beginning" >> $LOGFILE 2>&1

#MAIN
for ((u=0;u<=$USERNUM;u++))
do
	#log: user being sent to
	echo "$(date "+%m%d%Y %T"): To user: " "$USERNAME$u@$DOMAIN" >> $LOGFILE 2>&1
	#Create emails
	for (( i=1; i<=$LOOP; i++ ))
	do
		#Randomize a file so we have non-duplicates in Retain
		n=$RANDOM

		#Randomize body text
		shuf -n $n $INFILE > $OUTFILE
	
		#Get single block of text and suppress stderr
		dd if=$OUTFILE of=$BODY ibs=$BODYSIZE count=1 status=noxfer >& /dev/null

		#Randomize csv data
		shuf -n $n $INDATA > $SHUFDATA

		#Get single block of text and suppress stderr
		dd if=$SHUFDATA of=$OUTDATA ibs=$BODYSIZE count=1 status=noxfer >& /dev/null
		#make csv file with header
		cat $HEADER $OUTDATA > $ATTACHMENT

		#send email to user
		(cat $BODY)  | mail -s "localGoPostal,$n $(date)" -a $ATTACHMENT -a "$CODEFILE" $USERNAME$u@$DOMAIN
		#log: each email sent
		echo "$(date "+%m%d%Y %T")" "Item#:" "$i" "Item:" "$SUBJECT"  >> $LOGFILE 2>&1
	done
done
#log: When did it finish
echo "$(date "+%m%d%Y %T"): Done"
echo "$(date "+%m%d%Y %T"): Done" >> $LOGFILE 2>&1

exit $?  

Set this up with a cronjob so it runs every hour so you have a continuous flow of email. see Wikipedia [7] for details or generate a crontab [8]

[edit] CSV Source Material

Second, it is helpful if the messages have attachments. So something like a .CSV file is handy to create and attach as well. You can create a data file by hand, online [9] or with the following script. The results of the script can be downloaded here [10] You will also need to make a header file with 10 comma separated header fields [11]. Call it csvheader.dat

[edit] CSV Generator Script

 
#!/bin/bash
#csvdata.sh
# Stephan Fassmann 03/31/2016
# Create a file with lots of data for a base .csv file for shuffling to create lots of random files.
# Creates 10 comma separated fields per line.

#FILE (Update for your system)
FILELOCATION=/dev/csvdata

#Setup logging for progress tracking
LOGFILE=$FILELOCATION/csvdata.log
DATAFILE=$FILELOCATION/csvdata.dat

#CONSTANTS
LINEITEMS=10
LOOP=100000
DATASTART=0

#When did it start
echo "$(date "+%m%d%Y %T"): Beginning"
echo "$(date "+%m%d%Y %T"): Beginning" >> $LOGFILE 2>&1

#Create Array
for (( i=0; i<=$LOOP; i++ ))
	do
		DATAARRAY[$i]=$DATASTART","
		DATASTART=$((DATASTART+1))
	done

#Save Array
for (( j=0; j<=$LOOP; j++ ))
	do
		echo ${DATAARRAY[@]:j:$LINEITEMS} >> $DATAFILE 2>&1
		echo ${DATAARRAY[@]:j:$LINEITEMS} >> $LOGFILE 2>&1
	done

#When did it finish send code
echo "$(date "+%m%d%Y %T"): Done"
echo "$(date "+%m%d%Y %T"): Done" >> $LOGFILE 2>&1
exit $?  

[edit] Windows

[edit] Download Package

Download the Go-Postal.ps1 script package from [12] This has some more features than the script below, including multilingual html attachments, more interesting subject lines and logging.

Required: You need to specify your domain and the users the script should send to. There is an array of usernames, replace them with users that exist in your system.

Optional: The number of messages to send each time this script is run can be changed to more than 1. 4-5 per hour per user will provide average amounts of mail an average user receives. The message body size for each message (1023 bytes by default) can be set as well. Logging can be enabled or not. The log file size is limited to 10MB so it doesn't fill the server.

Otherwise, you can use the script below for a simpler email.

[edit] Text Source Material

First, you are going to need a source text. Something large enough to provide plenty of material but isn't copyrighted. I used this [13] but Project Gutenberg [14] has plenty of public domain material to choose from.

Save as *textdata.dat*

[edit] CSV Source Material

Second, it is helpful if the messages have attachments. So something like a .CSV file is handy to create and attach as well. You can create a data file by hand, online [15] or use the results of the script above can be downloaded here [16] You will also need to make a header file with 10 comma separated header fields [17]. Call it csvheader.dat

Save as *csvdata.dat*

[edit] Email Generator Script

This script uses different users then the bash script so you can have unique users in Retain. You can name them the same and use mailbox mapping.

#Go-Postal.ps1 
#Stephan Fassmann 03/31/2016
#You may have to set the execution policy to allow this script to run for just process: Set-ExecutionPolicy -Scope Process -ExecutionPolicy AllSigned
#This script expects to be run from C:\gopostal. If you want to run from a different folder you would need to change all instances.
#This script requires an input text file called input.dat. This can be any text file larger than 1kb.
#This script requires an input data file called csvdata.dat for the csv file body made up of 10 columns of comma separated data and many rows.
#This script requires an input text file called csvheader.dat for the csv file header made up of 10 columns of comma separated data with one header row.

#EDIT THESE FOR YOUR ENVIRONMENT
$smtpserver = "your.smpt.server.address"
$domainname = "your.domain.name"

#VARIABLES
#set test user range (test0-test9)
$username = "test"
$usernum = 0
$usermaxnum = 9
#number of messages to send each time, if set to send each hour 4 messages will provide an average (96) amount of email per user per day.
$loop = 4
#how big a message to build, default 10 lines
$bodysize = 10
$csvsize = 10
#$scriptDir = C:\gopostal
$scriptDir =  Split-Path $myInvocation.MyCommand.Path
#initialize variables
$randnum = 0
$i=1

#This script requires at least v4 of PowerShell
#Requires -Version 4.0
#"Starting Script"

Do{   
    Do{
        #Create Randomize data
		$randnum = Get-Random
        #Generate new randomized text file for body
        #"Creating body.tmp"
        #Shuffle the entire file and return the first bodysize lines
        $Idxs = 0..999
        Get-Content "$scriptDir\textdata.dat" -ReadCount 0 | ForEach {
            $sample = Get-Random -InputObject $Idxs -Count $bodysize 
            $_[$sample] | Add-Content "$scriptDir\body.tmp"
        }

        #Generate new randomized data file for csv attachment
        #"Creating shufdata.csv"
        #Shuffle the entire file and return the first csvsize lines
        $Idxs = 0..999
        Get-Content "$scriptDir\csvdata.dat" -ReadCount 0 | ForEach {
            $sample = Get-Random -InputObject $Idxs  -Count $csvsize 
            $_[$sample] | Add-Content "$scriptDir\shufdata.tmp"
        }

        #concatentate the header and shuffled body to create a randomized csv file attachment
        #"Creating shuffled.csv"
        Add-Content -Value (Get-Content $scriptDir\csvheader.dat,$scriptDir\shufdata.tmp) -Path $scriptDir\shuffled.csv

        #Send Randomized Message
        #"Sending Randomized Message"
        #Set body to randomized text
        $body = Get-Content -Path "$scriptDir\body.tmp" -Raw 
        
        #How long does it take to run (~250ms/send)
        #Measure-Command{Send-MailMessage -To "test$usernum@sf.gwava.net" -From "script@sf.gwava.net" -Subject "Go-Postal $randnum" -SmtpServer 10.1.4.213 -Body $body -Attachments "C:\Users\Administrator\dev\shuffled.csv"
        #}
        #Send message with randomized body, and csv attachment and script attachment so make sure it handles non-unique attachments, plus simple version control, add date to the subject line so we can see if there are flag errors more easily, plus random number in subject to make it unique
        Send-MailMessage -To "$username$usernum@$domainname" -From "script@$domainname" -Subject "Go-Postal $(Get-Date) $randnum" -SmtpServer ($smtpserver) -Body $body -Attachments "$scriptDir\shuffled.csv"
        #"Randomized Message Sent to:test$usernum"
        #increment the email loop counter
        $i++

        #Delete randomized data
        #Remove shufdata.tmp
        #"Deleting shufdata.tmp"
        Remove-Item "$scriptDir\shufdata.tmp"
        #Remove shuffled.csv
        #"Deleting shuffled.csv"
        Remove-Item "$scriptDir\shuffled.csv"
        #Remove body.tmp
        #"Deleting body.tmp"
        Remove-Item "$scriptDir\body.tmp"

    } While ($i -le $loop)
    #increment the user loop counter
    $usernum++
    $i=1
} While($usernum -le $usermaxnum)

[edit] Task Scheduler

You can use Task Scheduler to automate sending the messages. Schedule the script to run hourly. Run powershell.exe with the option set to the script.

[edit] Notes on Exchange

On a test system, Exchange can be limited in performance. So limit the number of messages sent to only a few a second. On my VM this script will send about 4 messages per second which Exchange can keep up with. You can use a Linux VM and the bash scripts to send to Exchange but that sent 20 messages a second and that overwhelmed the Exchange server.

Return to Retain Training