MSM Data Collector - User Guide

Description	End-user guide for the MSM Data Collector Client
Author(s)	Cloud Solution Team

MSM Data Collector - User Guide

Welcome to the MSM Data Collector Client user guide. This tool helps you extract telemetry data from your MSM server and upload it to the Cloud Solution for analysis.

What is the MSM Data Collector?

The MSM Data Collector is a desktop application that connects to an MSM, extracts telemetry data from the database and uploads it securely to the Cloud Solution for exploring, monitoring and analytics.

How It Works

1. Configure your MSM server connection details
2. Choose your preferred operation mode
3. Start the data extraction process
4. Monitor progress in real-time through the interface

Getting Help

If you encounter issues:

• Check the Troubleshooting section
• Review the FAQ for common questions
• Contact your system administrator for technical support

Quick Start

New to the Data Collector? Start here:

1. Install the application
2. Configure your connection
3. Run your first extraction

Getting Started

Installation

This guide will help you install the MSM Data Collector Client on your computer.

System Requirements

Before installing, ensure your system meets these requirements:

• Operating System: Windows 10/11, macOS, or Linux
• Python: Version 3.11 or higher
• Network: Access to your MSM server
• Storage: At least 500 MB free space

Installation Steps

Step 1: Install Python

If you don't have Python 3.11+ installed:

WindowsmacOSLinux

1. Download Python from python.org
2. Run the installer
3. Important: Check "Add Python to PATH" during installation
4. Click "Install Now"

brew install python@3.11

sudo apt update
sudo apt install python3.11 python3-pip

Step 2: Verify Python Installation

Open a terminal/command prompt and run:

python --version

You should see Python 3.11.x or higher.

Step 3: Download the Data Collector

1. Download the latest version from your administrator
2. Extract the ZIP file to a location on your computer (e.g., C:\MSMDataCollector or /home/user/MSMDataCollector)

Step 4: Install Dependencies

Open a terminal/command prompt in the extracted folder and run:

pip install -r requirements.txt

This will download and install all necessary components.

Installation Complete

You're now ready to configure the application. Proceed to Configuration.

Troubleshooting Installation

Python not found

If you get "python is not recognized" error:

• Windows: Make sure you checked "Add Python to PATH" during installation
• macOS/Linux: Try using python3 instead of python

Permission errors during pip install

Run the command with user flag:

pip install --user -r requirements.txt

Still having issues?

Contact your system administrator for assistance.

Configuration

Before using the Data Collector, you need to configure it with your MSM server connection details.

Getting Your Configuration Details

Contact your system administrator to obtain:

• MSM server hostname/IP address
• SSH username and password
• Database connection details
• AWS credentials (if applicable)

Configuration File

The application uses a config.yaml file for settings.

Creating the Configuration File

1. In the application folder, create a new file named config.yaml
2. Copy the template below and fill in your details

Configuration Template

msm:
  host: "your-msm-server-hostname"
  ssh_port: 22
  ssh_user: "your-username"
  ssh_password: "your-password"
  postgres_host: "localhost"
  postgres_port: 5432
  postgres_user: "postgres"
  postgres_password: "database-password"
  postgres_db: "msmdb"
  table_name: "brief_controller_telemetry"
  timestamp_column: "msm_timestamp"

api:
  base_url: "https://data-collector.aws.ayk-services.com"

logging:
  level: "INFO"
  file: "logs/extraction.log"

Configuration Fields Explained

MSM Section

Field	Description	Example
host	MSM server hostname or IP	msm-server.company.com
ssh_port	SSH port (usually 22)	22
ssh_user	Your SSH username	admin
ssh_password	Your SSH password	SecurePass123
postgres_host	Database host (usually localhost)	localhost
postgres_port	Database port (usually 5432)	5432
postgres_user	Database username	postgres
postgres_password	Database password	dbPass456
postgres_db	Database name	msmdb
table_name	Data table name	brief_controller_telemetry
timestamp_column	Timestamp field name	msm_timestamp

API Section

Field	Description
base_url	Cloud platform URL (provided by administrator)

Logging Section

Field	Description
level	Log detail level (INFO recommended)
file	Where to save logs

Security Notice

The config.yaml file contains sensitive passwords. Keep it secure and never share it with unauthorized users.

Verifying Configuration

After creating your config.yaml file:

1. Save the file in the application root folder
2. Proceed to Starting the Application

Configuration Tips

Keep a Backup

Save a copy of your config.yaml file in a secure location. You'll need it if you reinstall the application.

Test Configuration

Ask your administrator about test credentials to verify your setup before using production data.

Authentication

To use Online Mode or Offline Upload Mode, you need to authenticate with your AYK credentials using Microsoft Authentication.

When Authentication is Required

Authentication is required for:

• ✅ Online Mode - Uploading data in real-time
• ✅ Offline Upload Mode - Uploading previously extracted files

Authentication is not required for:

• ❌ Offline Mode - Only extracts data locally, no upload

How to Log In

Step 1: Click Login Button

From the mode selection screen, click the "Login" button.

Step 2: Note the Authentication Code

A code will be displayed in the application window. Keep this code visible - you'll need it in the next step.

Step 3: Open Browser Authentication Page

A browser window will automatically open with the Microsoft authentication page.

Step 4: Enter the Code

Enter the authentication code from Step 2 into the browser page and click "Next".

Step 5: Select Your Account

Choose the AYK account you want to use for the data collector.

Step 6: Confirm Login

Click "Continue" to confirm you want to log in.

Step 7: Verification Complete

After successful authentication, return to the application. Online Mode and Offline Upload Mode should now be available.

Session Duration

Your authentication session will remain active until you explicitly log out

Security Notes

Keep Your Session Secure

• Don't share your authentication code with others
• Always log out on shared computers
• Close the application when not in use
• Report any suspicious authentication requests to IT

Logging Out

To log out:

1. Return to the mode selection screen
2. Click the "Logout" button
3. Your session will be terminated

You'll need to authenticate again to use Online Mode or Offline Upload Mode.

Usage Guide

Starting the Application

This guide shows you how to start and access the MSM Data Collector interface.

Launching the Application

Step 1: Open a Terminal

Navigate to the application folder:

WindowsmacOS/Linux

1. Open File Explorer
2. Navigate to your installation folder
3. Type cmd in the address bar and press Enter

cd /path/to/MSMDataCollector

Step 2: Start the Application

Run the following command:

python run_tkinter_ui.py

A desktop window will open with the MSM Data Collector interface.

Application Started

You should see a window with the MSM Data Collector interface and options for different operation modes.

Understanding the Interface

Mode Selection Screen

When you first launch the application, you'll see the mode selection screen:

The main interface allows you to choose between different operation modes:

• Online Mode - Extract and upload data in real-time (requires authentication)
• Offline Mode - Extract data to files for later upload (no authentication required)
• Upload Mode - Upload previously extracted files (requires authentication)

Authentication

To use Online Mode or Upload Mode, you need to authenticate with your AYK credentials.

Authentication Required

See the Authentication Guide for detailed login instructions.

Quick steps: 1. Click the "Login" button 2. Note the code displayed 3. Enter the code in the browser window that opens 4. Select your AYK account and confirm

After successful authentication, all modes will be available:

Selecting a Mode

After selecting a mode, you'll see controls to configure extraction parameters such as date ranges and batch sizes, along with progress monitoring.

Stopping the Application

To stop the application:

1. Close the application window using the X button or File menu
2. If the window becomes unresponsive, you can close the terminal window

Don't Force Close During Extraction

Avoid closing the application during active extraction. Use the Stop button in the interface first, then close the window.

Next Steps

Choose your operation mode:

• Online Mode - For stable connections
• Offline Mode - For unstable connections
• Upload Mode - To upload previously extracted data

Troubleshooting

Application Won't Start

1. Verify Python is installed: python --version
2. Check dependencies are installed: pip install -r requirements.txt
3. Verify config.yaml exists in the application folder
4. Check that tkinter is installed:
5. Windows/Mac: Should come with Python
6. Linux: sudo apt-get install python3-tk

Window Doesn't Appear

1. Check the terminal for error messages
2. Verify tkinter is available: python -c "import tkinter"
3. Try running with: python -u run_tkinter_ui.py

Application Freezes

1. Check the terminal for error messages
2. Close the window and restart the application
3. Review logs in logs/extraction.log

Online Mode

Online Mode extracts data from your MSM server and uploads it to the cloud platform in real-time. This is the recommended mode when you have a stable internet connection.

When to Use Online Mode

Use Online Mode when:

• ✅ You have a stable internet connection
• ✅ You want real-time data upload
• ✅ You need immediate data availability in the cloud platform
• ✅ You have reliable access to the MSM server

How Online Mode Works

1. Connects to your MSM server via SSH
2. Extracts telemetry data from the database
3. Immediately uploads data to the cloud platform
4. Processes data in batches for efficiency

Using Online Mode

Step 1: Select Online Mode

In the application interface, select "Online Mode" from the mode selection.

Step 2: Set Date Range

Configure the extraction period:

1. Start Date & Time: The beginning of your data range
2. End Date & Time: The end of your data range

Choosing Date Ranges

• For initial extraction: Start with a small date range (e.g., 1 day) to verify everything works
• For regular extractions: Use the last extraction time as your start date
• For historical data: Process in chunks of 7-30 days to avoid timeouts

Choosing Date Ranges

• The server keeps track of all the information sent, if a day has already been uploaded within your date range, the server will tell the Data Collector to avoid fetching it from the MSM.
• This means that, for certain date ranges, you may experience less uploaded days than the currently selected ones, and that is fine by design.

Step 3: Configuration

Before starting the extraction, you need to configure the connection settings to your MSM server. The application provides two collapsible configuration panels in the sidebar.

SSH Configuration

Click to expand the SSH Configuration panel and enter your MSM server details:

• Host: The IP address or hostname of your MSM server
• Port: SSH port (default: 22)
• User: SSH username for authentication
• Password: SSH password for authentication

SSH Authentication

The application currently supports password-based SSH authentication only.

Database Configuration

Click to expand the Database Configuration panel and enter your PostgreSQL database details:

• Host: Database host (typically localhost when accessed via SSH tunnel)
• Port: PostgreSQL port (default: 5432)
• User: Database username
• Password: Database password
• Database: Database name (default: msmdb)
• Table: Table containing telemetry data (default: brief_controller_telemetry)
• Timestamp Column: Column name for timestamps (default: msm_timestamp)

Default Values

The default values are pre-configured, they should be valid for all implementations.

Validation

After entering your configuration:

1. Click the "Test Configuration" button
2. The application will:
   • Test the SSH connection to your MSM server
   • Auto-detect the MSM hostname
   • Test the database connection via SSH tunnel
   • Validate that the specified table and columns exist
3. Wait for the validation to complete (typically 10-15 seconds)
4. Look for the "✅ Configuration valid" message

Start Button Disabled

The Start button will remain disabled until the configuration is successfully validated. You must complete validation before you can begin extraction.

Connection Cleanup

After validation completes, the application takes a few seconds to properly close test connections before enabling the Start button. This ensures no lingering connections interfere with the actual extraction.

Process History

After validation completes, all uploads that were done for the current computer will be shown on the Process History section.

Step 4: Start Extraction

1. Click the "Start Online Extraction" button
2. Monitor the progress in real-time

Step 5: Monitor Progress

The interface displays:

• Records Processed: Total records extracted
• Current Batch: Current batch being processed
• Upload Status: Success/failure of uploads
• Errors: Any issues encountered

Process Management

Understanding Process Restrictions

One Process Per MSM

CRITICAL: Only ONE active process can exist for each MSM at any given time. This is enforced by the system to prevent:

• Data duplication and conflicts
• Resource contention on the MSM server
• Overlapping extraction ranges
• Inconsistent state in the cloud platform

If you attempt to start a new extraction while a process is still running, stopped, or in a non-completed state, you must either:

1. Wait for the current process to complete
2. Restart the existing process (continues from where it left off)
3. Abort the existing process (permanently cancels it)

Process States

Each process can be in one of the following states:

State	Icon	Description	Actions Available
Running	🔵 Blue	Extraction is actively processing	Stop
Stopped	🟡 Yellow	User stopped the process mid-extraction	Restart, Abort
Completed	🟢 Green	All partitions successfully processed	None (view only)
Failed	🟡 Yellow	Process encountered an error	Restart, Abort
Aborted	🔴 Red	Process was permanently cancelled	None (view only)

Managing Stopped Processes

When you stop an extraction (using the Stop button), the process enters a STOPPED state:

You have two options:

Option 1: Restart Process

Click "Restart Process" to continue the extraction from where it stopped:

• Preserves all progress made so far
• Skips partitions that were already completed
• Continues with remaining partitions
• Uses the same date range as the original process
• Maintains the same process ID

Recommended for Interrupted Extractions

If an extraction was interrupted due to network issues, accidental stop, or other temporary problems, Restart is the best option to resume without losing progress.

Option 2: Abort Process

Click "Abort Process" to permanently cancel the process:

• Marks the process as ABORTED (cannot be restarted)
• Frees up the MSM to start a new extraction
• All uploaded data remains in the cloud platform
• Incomplete partitions will need to be re-extracted in a new process

Use Abort Sparingly

Only abort a process if:

• You want to completely abandon this extraction
• You need to start a different date range
• The process is stuck and cannot be restarted

Warning: Aborting loses all progress tracking. You'll need to start fresh with a new extraction.

Process History

The Process History panel shows all past and current processes for the connected MSM:

• Automatically updates when processes are created, stopped, or completed
• Shows process status, date range, and completion percentage
• Displays creation and completion timestamps
• Provides activity logs for each process
• Available actions depend on process state

Viewing Process Details

Click on any process in the history to expand it and view:

• Detailed activity log
• Partition completion status
• Creation and finish timestamps
• Available actions (Restart/Abort buttons)

Best Practices

Do

• Keep the application window open
• Monitor progress regularly
• Note any error messages
• Wait for "Extraction Complete" message

Don't

• Close the application window during extraction
• Start another extraction for the same MSM simultaneously
• Force close the application (use Stop button instead)

After Extraction

Success

When extraction completes successfully:

• You'll see "Extraction Complete" message
• All data has been uploaded to the Cloud Solution, and will start becoming available in the coming minutes.
• Check the logs for detailed statistics

Resuming Incomplete Extraction

If an extraction was interrupted:

1. Note the last successfully processed timestamp from logs
2. Start a new extraction with that timestamp as the start date
3. This prevents duplicate data

Troubleshooting Online Mode

Connection Lost During Extraction

If you lose connection:

• The application will show connection errors
• Note the last successful batch
• Stop the extraction and retry from that point

Slow Performance

If extraction is very slow:

• Check your network connection speed
• Consider using Offline Mode instead

Authentication Errors

If you see authentication failures:

• Check with your administrator that your account is active
• Ensure you have network access to the MSM server

Upload Failures

If uploads fail repeatedly:

• Check your internet connection
• Check with your administrator about the Cloud Solution status
• Consider switching to Offline Mode

Offline Mode

Offline Mode extracts data from your MSM server and saves it to compressed files on your computer.

You can upload these files later when you have a better connection.

When to Use Offline Mode

Use Offline Mode when:

• ✅ You have an unstable internet connection
• ✅ You want to extract data now and upload later

How Offline Mode Works

1. Connects to your MSM server via SSH
2. Extracts telemetry data from the database
3. Saves data to compressed CSV files on your computer
4. Upload files later using Offline Upload Mode

Using Offline Mode

Step 1: Select Offline Mode

In the application interface, select "Offline Mode" from the mode selection.

Step 2: Set Date Range

Configure the extraction period:

1. Start Date & Time: The beginning of your data range
2. End Date & Time: The end of your data range
3. Force re-extract existing partitions: To re-extract already extracted partitions, as the Data Collector keeps track of all extracted partitions and will not re-extract an already extracted partition, unless this is checked.

Force Re-extract

Only enable this option if you need to re-extract data for dates you've already processed. This will overwrite existing files.

Step 3: Configuration

Offline Mode uses the same configuration as Online Mode. You need to configure:

• SSH Configuration: Connection details to your MSM server
• Database Configuration: Database settings
• Validation: Test your configuration before extraction

See the Online Mode Configuration Guide for detailed instructions on:

• SSH connection setup
• Database configuration
• Validation process

Step 4: Start Extraction

1. Click the "Start Offline Extraction" button
2. Monitor the progress

Step 5: Monitor Progress

The interface displays:

• Records Processed: Total records extracted
• Files Created: Number of CSV files saved
• Storage Used: Disk space consumed
• Current Status: Active extraction information

Storage Location

All extracted files are saved to:

offline_extractions/
├── extraction_20240101_120000_part_001.csv.gz
├── extraction_20240101_120000_part_002.csv.gz
└── ...

Disk Space

Monitor available disk space. As a rule of thumb, expect ~3 MB per day per SSC on the MSM.

Next Steps

When you're ready to upload:

1. Ensure you have a stable internet connection
2. Use Upload Mode to upload your files
3. Keep the files until upload is confirmed successful

Advantages of Offline Mode

✅ Resilient: Network issues don't interrupt data extraction

✅ Backup: You have local copies of your data

✅ Flexible: Upload when convenient

✅ Reliable: Less chance of data loss

Disadvantages of Offline Mode

❌ Two-step process: Extract then upload separately

❌ Storage required: Need local disk space

❌ Delayed availability: Data not immediately available in cloud

Upload Mode

Upload Mode allows you to upload previously extracted offline files to the cloud platform. Use this after you've used Offline Mode to extract data.

When to Use Upload Mode

Use Upload Mode when:

✅ You now have a stable internet connection

✅ You have extracted data in Offline Mode

✅ You need to upload files to the Cloud Solution

How Upload Mode Works

1. Reads the database for available offline extractions
2. Groups partitions by MSM and date ranges
3. Displays MSMs with pending uploads
4. Uploads selected files to the Cloud Solution
5. Tracks upload progress and status

Using Upload Mode

Step 1: Select Upload Mode

In the application interface, select "Upload Mode" from the mode selection.

Step 2: Select MSM

The interface displays all MSMs that have offline data available for upload.

• MSM and pending count: The MSM hostname and number of pending uploads
• Total Partitions: Total number of extracted partitions
• Pending Upload: Number of partitions not yet uploaded
• Already Uploaded: Number of partitions successfully uploaded
• Total Size: Total size of all partition files

No Data Available

If no MSMs are shown, you need to first extract data using Offline Mode.

Step 3: View Partitions

Click on an MSM card to expand it and view the Partitions tab, which shows:

• Date: The partition date
• File Path: Location of the extracted file
• Status: Upload status (Pending, Uploaded, Failed)
• Size: File size

You can:

• Select specific partitions using checkboxes to delete them
• Delete Selected: Remove selected partition files from disk
• View upload history in the History tab

Deleting Partitions

Deleting a partition removes the file from disk permanently.

Only delete partitions that:

• Have already been uploaded
• Are no longer needed
• Are conflicting.

Step 4: Start Upload

1. Click on "Upload Pending"
2. The upload progress screen will open

The upload screen shows:

• Date Ranges: Organized view of partitions being uploaded
• Progress Bar: Overall upload progress
• Live Logs: Real-time upload status messages

Step 6: Monitor Progress

During upload, the interface displays:

• Status: Current operation (Authenticating, Uploading, etc.)
• Process ID: Unique identifier for this upload process
• Partitions Progress: Number of partitions completed vs total
• Current Partition: The date being uploaded
• Live Logs: Detailed upload activity

Upload Complete

When upload completes successfully, all partitions will be marked as uploaded in the database and data will be available in the Cloud Solution.

Process Management

Upload Mode uses the same process management system as Online Mode:

• One process per MSM at a time
• Ability to Stop, Restart, or Abort processes
• Process History shows all past and current uploads

See the Online Mode Process Management Guide for detailed information on:

• Understanding process restrictions
• Process states (Running, Stopped, Completed, Failed, Aborted)
• Managing stopped processes
• Restart vs Abort decisions

Best Practices

Do

• Keep the application window open during upload
• Ensure stable internet connection
• Monitor the live logs for errors
• Wait for "Upload Complete" message before closing

Don't

• Close the application window during upload
• Delete partition files before confirming successful upload
• Start multiple uploads for the same MSM simultaneously

After Upload

Successful Upload

When upload completes successfully:

• ✅ All partitions marked as "Uploaded" in database
• ✅ Data available in the Cloud Solution (will start processing within minutes)
• ✅ Process marked as "Completed" in Process History
• ✅ Safe to delete uploaded files if desired

Managing Files

After successful upload, you can:

1. Keep files: Retain local copies as backup
2. Delete files: Free up disk space by deleting uploaded partitions using the "Delete Selected" feature in the Partitions tab

Keeping Backups

Consider keeping uploaded files for a few days until you verify the data appears correctly in the Cloud Solution.

Troubleshooting

This page helps you resolve common issues with the MSM Data Collector Client.

Application Won't Start

Python Not Found

Symptom: Error message "python is not recognized as a command"

Solution:

1. Verify Python is installed: Open terminal and run python --version
2. If not installed, see Installation Guide
3. On Windows, ensure "Add Python to PATH" was checked during installation
4. Try using python3 instead of python (macOS/Linux)

Missing Dependencies

Symptom: Import errors or module not found errors

Solution:

pip install -r requirements.txt

If that fails, try:

pip install --user -r requirements.txt

Tkinter Not Available

Symptom: "tkinter module not found" error

Solution:

• Windows/Mac: Tkinter should come with Python. Reinstall Python if missing.
• Linux: Install tkinter package:

  sudo apt-get install python3-tk
  

Connection Issues

Can't Connect to MSM Server

Symptom: SSH connection failed or timeout errors

Solution:

1. Verify MSM server is online and accessible
2. Check config.yaml has correct host, port, username, and password
3. Test SSH connection manually:

   ssh username@msm-server-hostname
   

4. Verify firewall allows SSH connections (port 22)
5. Contact your network administrator if issues persist

Database Connection Failed

Symptom: PostgreSQL connection errors

Solution:

1. Verify SSH tunnel to MSM server is working
2. Check config.yaml database credentials are correct
3. Verify database is running on the MSM server
4. Check database name, username, and password
5. Contact your database administrator

Upload Failures

Symptom: Can't upload to cloud platform

Solution:

1. Check your internet connection
2. Verify api.base_url in config.yaml is correct
3. Test API accessibility:

   ping data-collector.aws.ayk-services.com
   

4. Check with administrator about cloud platform status
5. Try Offline Mode then upload later

Performance Issues

Extraction is Very Slow

Symptom: Data extraction takes much longer than expected

Solutions:

1. Reduce batch size to 500 or 250
2. Check your network connection speed
3. Verify MSM server is not under heavy load
4. Use Offline Mode for better performance
5. Extract data in smaller date ranges

Application Uses Too Much Memory

Symptom: Computer becomes slow or unresponsive

Solutions:

1. Reduce batch size in extraction settings
2. Close other applications
3. Extract data in smaller date ranges
4. Restart the application between large extractions

Slow Upload Speed

Symptom: File uploads take very long

Solutions:

1. Check internet connection speed
2. Upload during off-peak hours
3. Use wired connection instead of WiFi
4. Close bandwidth-heavy applications
5. Upload smaller files or split large extractions

Data Issues

No Data Extracted

Symptom: Extraction completes but zero records

Solutions:

1. Verify date range has data in the database
2. Check table_name in config.yaml is correct
3. Verify timestamp_column matches database schema
4. Check database permissions for your user account
5. Review logs at logs/extraction.log

Duplicate Data

Symptom: Same data uploaded multiple times

Solutions:

1. Don't restart extractions from the same start date
2. Track last successful extraction timestamp
3. Use that timestamp as start date for next extraction
4. Check logs to find last processed record
5. Contact administrator to remove duplicates from cloud

Missing Data

Symptom: Some records not extracted

Solutions:

1. Check logs for error messages during extraction
2. Verify date range covers intended period
3. Check for interruptions during extraction
4. Re-run extraction for missing date ranges
5. Verify database had data for that period

Configuration Issues

Invalid Configuration File

Symptom: Error loading config.yaml

Solutions:

1. Verify YAML syntax is correct (use a YAML validator)
2. Check indentation (use spaces, not tabs)
3. Ensure all required fields are present
4. Remove any special characters from passwords
5. Compare with the configuration template

Authentication Failed

Symptom: Invalid credentials errors

Solutions:

1. Double-check username and password in config.yaml
2. Verify there are no extra spaces before/after credentials
3. Check if password contains special characters that need escaping
4. Confirm account is active with your administrator
5. Test credentials manually via SSH

File and Storage Issues

Can't Create Output Files

Symptom: Permission denied errors when saving files

Solutions:

1. Check you have write permissions to the application folder
2. Create offline_extractions/ folder manually if missing
3. Run application with appropriate permissions
4. Choose a different output directory with write access

Disk Space Full

Symptom: Not enough space errors

Solutions:

1. Free up disk space
2. Delete old extraction files after successful upload
3. Extract smaller date ranges
4. Use different drive with more space
5. Clean up log files in logs/ folder

Can't Delete Files

Symptom: Permission errors when deleting files

Solutions:

1. Close the application first
2. Ensure no other program has files open
3. Check file permissions
4. Delete files manually from file explorer

Interface Issues

Application Window Not Appearing

Symptom: Command runs but no window appears

Solutions:

1. Check terminal for error messages
2. Verify tkinter is installed: python -c "import tkinter"
3. Check if window appeared behind other windows
4. Try running with elevated permissions
5. Check display settings (especially on Linux with remote sessions)

Interface Frozen

Symptom: Window doesn't respond to clicks

Solutions:

1. Check terminal for errors
2. Wait - large operations take time
3. Close and restart the application
4. Check logs at logs/extraction.log

Progress Not Updating

Symptom: Progress bar stuck or not moving

Solutions:

1. Wait - large batches take time
2. Check terminal for error messages
3. Check logs at logs/extraction.log
4. Restart extraction if genuinely stuck

Error Messages

"Connection refused"

Cause: Can't reach the server

Solution: Check network connection, firewall settings, and server address

"Authentication failed"

Cause: Invalid credentials

Solution: Verify username and password in config.yaml

"Permission denied"

Cause: Insufficient file system or database permissions

Solution: Check file permissions or contact administrator for database access

"Timeout error"

Cause: Operation took too long

Solution: Reduce batch size, check network connection, or split into smaller extractions

"File not found"

Cause: Missing required file

Solution: Verify file path, check file exists, or recreate from template

Getting Additional Help

If you can't resolve your issue:

Check the Logs

Detailed logs are in:

logs/extraction.log

Look for ERROR or WARNING messages.

Information to Provide

When contacting support, include:

• Error message (exact text)
• What you were doing when error occurred
• Relevant log entries from logs/extraction.log
• Your Python version: python --version
• Operating system and version
• Screenshot of the error (if applicable)

Contact Support

• Technical Issues: Contact your system administrator
• Data Questions: Contact your data team
• Access Issues: Contact your IT support team

Frequently Asked Questions

See the FAQ page for common questions about usage and features.

Frequently Asked Questions

General Questions

What is the MSM Data Collector?

The MSM Data Collector is a tool that extracts telemetry data from your MSM server and uploads it to a cloud platform for monitoring, analytics, and reporting.

Do I need to be a programmer to use it?

No. The Data Collector has a user-friendly desktop interface that requires no programming knowledge. If you can use basic desktop applications, you can use this tool.

Is my data secure?

Yes. All data transfers use encrypted SSH and HTTPS connections. Your configuration file contains sensitive credentials and should be protected accordingly.

Installation and Setup

What do I need to install?

You need Python 3.11 or higher and the application files. All dependencies are installed automatically. See the Installation Guide for details.

Where do I get my configuration details?

Contact your system administrator for MSM server connection details, database credentials, and cloud platform URL.

Can I run this on any computer?

Yes. The Data Collector works on Windows, macOS, and Linux. You need Python 3.11+ and network access to your MSM server.

How much disk space do I need?

For the application: ~100 MB For offline extractions: ~1 MB per 10,000 records extracted

Can multiple people use the same installation?

Each user should have their own installation with their own configuration file for security and audit purposes.

Usage Questions

Which mode should I use?

• Stable internet connection: Use Online Mode
• Unstable connection or large dataset: Use Offline Mode
• Previously extracted files: Use Upload Mode

How long does an extraction take?

It depends on: - Amount of data - Network speed - Server performance - Mode used (online vs offline)

Typical: 5,000-10,000 records per minute in Online Mode

Can I run extractions in the background?

The application must stay open during extraction, but you can minimize the application window. Don't close the window or terminal completely.

Can I stop an extraction and resume later?

Yes. Use the Stop button, note the last processed timestamp from logs, and start a new extraction from that point.

How often should I run extractions?

This depends on your data collection needs. Common schedules: - Daily for regular monitoring - Weekly for less critical data - On-demand for specific analyses

Can I extract data from multiple MSM servers?

You'll need separate configuration files for each server. Stop the current extraction before switching to a different server configuration.

Data Questions

What data is extracted?

The Data Collector extracts telemetry data from your MSM database's telemetry table. The specific data depends on your MSM configuration.

How far back can I extract data?

You can extract data from any date in your MSM database. Older data may take longer to extract depending on database size.

Will extraction affect my MSM server performance?

Extraction uses minimal resources. The application reads data through SSH tunnel with configurable batch sizes to minimize impact.

Can I extract data while MSM is recording?

Yes. The extraction doesn't interfere with active data recording on your MSM server.

What if I accidentally extract the same data twice?

The system doesn't automatically detect duplicates. Always track your last extraction date to avoid re-extracting data.

How do I know what date range to extract?

• For first extraction: Start from your earliest needed date
• For subsequent extractions: Use the end date from your last extraction as the start date
• Check extraction logs for last processed timestamps

Technical Questions

What database does MSM use?

MSM uses PostgreSQL. The Data Collector connects via SSH tunnel to ensure secure access.

What format is the extracted data?

Data is extracted in CSV format and compressed with gzip (.csv.gz files) for efficient storage and transfer.

Where are files stored?

• Offline extractions: offline_extractions/ folder
• Logs: logs/ folder
• Configuration: config.yaml file in application root

Can I customize the extraction?

Basic customization (date ranges, batch sizes) is available in the interface. Advanced customization requires editing the configuration file.

What happens if my computer crashes during extraction?

In Online Mode: Note the last successful batch from logs and restart from that point. In Offline Mode: Completed files are safe. Restart extraction for remaining data.

Does this work with VPN?

Yes, as long as your VPN allows access to the MSM server and cloud platform.

Troubleshooting

Why is extraction so slow?

Common causes: - Large date range - Slow network connection - High server load - Large batch sizes

Try reducing batch size or using Offline Mode.

Why do I get authentication errors?

• Check username and password in config.yaml
• Verify account is active
• Ensure no typos or extra spaces in credentials

What if uploads keep failing?

1. Verify internet connection
2. Check cloud platform URL in config
3. Try Offline Mode then upload later
4. Contact administrator about cloud platform status

Where are error logs?

Check logs/extraction.log for detailed error information.

Can't find offline extraction files?

Check the offline_extractions/ folder in your application directory. Files are named extraction_YYYYMMDD_HHMMSS_part_NNN.csv.gz.

Best Practices

Should I delete old extraction files?

Yes, but only after confirming successful upload to the cloud platform. Keep files until you verify data is accessible in the cloud.

How do I avoid extracting duplicate data?

Always use the end timestamp of your previous extraction as the start timestamp for your next extraction.

What's the best batch size?

• Fast connection (50+ Mbps): 2000-5000
• Medium connection (10-50 Mbps): 1000-2000
• Slow connection (<10 Mbps): 500-1000

Should I use Online or Offline Mode?

• Online Mode: Best for small-to-medium datasets with stable internet
• Offline Mode: Best for large datasets or unreliable connections

How do I backup my configuration?

Copy your config.yaml file to a secure location. Keep it updated when credentials change.

Security Questions

Is my password visible in config.yaml?

Yes, credentials are stored in plain text in config.yaml. Protect this file with appropriate file permissions and don't share it.

Can others see my data?

Only if they have access to your computer and the application folder. Use appropriate file permissions to protect sensitive data.

Should I share my config.yaml file?

No. Never share your configuration file as it contains passwords and sensitive connection details.

What happens to my data after upload?

Data is stored securely in the cloud platform where authorized users can access it for monitoring and analytics. Contact your administrator about data retention policies.

Updates and Maintenance

How do I update to a new version?

Your administrator will provide new version files. Back up your config.yaml, replace application files, and restore your configuration.

Do I need to update Python?

Only if a new version requires it. Check release notes from your administrator.

How do I know if there's a new version?

Your administrator will notify you of important updates. There's no automatic update check in the application.

Still Have Questions?

If your question isn't answered here:

1. Check the Troubleshooting guide
2. Review the specific mode documentation:
3. Online Mode
4. Offline Mode
5. Upload Mode
6. Contact your system administrator for technical support