technical

Network

  • tcptrack
  • lsof -i TCP
  • tcpdump
  • wireshark
  • tshark

Process

- top
- free
- dtop
- strace

Disk

  • iostat
  • iostat -x 1

Strace

To follow a process

strace -p 3569

Summary of a long running process

strace -c -p 3569

Run Process and grab the pid

 my-app &
 pid=$!
 strace -p $pid

Monitoring

  • using iperf
  • Ganglia
  • nify
    • Data Gathering
  • node Red
    • Data Gathering
  • Python
    • scapy
    • Packet Disassembler
  • Wireshark
  • nc
  • i2 with PCAP formatted data

Production

  • Dashboards
    • d3
  • ganglia
    • gmond custom identifiers
  • Logs
    • python
    • use daquari
    • OS
    • /var/log
    • Share Knowledge
    • Wikipedia
      • Usually globally
      • Sometimes per project
    • Markdown
    • Documentation
    • Manuals
    • How-To guides
    • Cookbooks
    • User training Videos
    • ElasticSearch over the Content store for easy of search

Programming

  • Python
    • Python3 not 2
    • use Virtual Environment always
    • Internal PYPI Store
    • Lots of VM's in production and several used in Development process.
  • C++
    • g++
  • C#
    • Limited now
  • Hadoop
    • HBase
    • happybase
    • Custom Loaders and readers
    • Hive
    • Pheonix
    • HDP HBase add on
    • Spark
    • Limited expierience
    • Real Time GPS / Shopping model
    • Flume
    • Moving Log files into HDFS
    • Mahout
    • Processing of PDF, Excel etc into HDFS
    • Scoop
    • Read in SQL Databases into a Hadoop env
  • Problem Solving
    • Replicate the issue (safely)
    • Confirm the issue
    • Since When ?
    • Whats been recently changed
    • Have I been working on this ?
    • Compare the Code Repository
    • Changes ?
    • Why
    • What in the Env has changed ?
    • Machine
    • Software
    • Application
    • OS
    • Framework
    • Libraries
    • Can I run the code in a Dev environment ?
    • Debugging
    • Logging
    • Find the Records.

System Services

  • initctl list
  • service --status-all
  • initctl show-config myservice

Run Levels

Usually defined in /etc/inittab

  • 0 Shutdown
  • 1 SingleUser Mode - No Network
  • 2 Multi User - No Network
  • 3 Multi User with Network. Normal
  • 4 Special
  • 5 Same as 3 - but with Display manager
  • 6 reboot

Get the Run Level

runlevel

It returns

N 5

Which means No Previous, Runlevel 5

To see the Previous Run-Level, do an prevlevel command.

On Ubuntu

chkconfig

Bash

Interactive

It executes the 1st readable version of

  • ~/.bash_profile
  • ~/.bash_login
  • ~/.profile,

Batch

  • ~/.bashrc

Software

Code Comprehension

  • Code should be easy to comprehend
    • not always possible
    • Some is difficult / complex
  • Documentation inside code
    • Modern languages enforce this to certain degrees

Bug Identification

  • Severity
  • Priority
  • Software releases
  • Types
    • Arithmetic
    • rounding
    • Logic
    • Sometime difficult if using lots of bitwise operations
    • Syntax
    • Usually resolved in compiled languages.
    • Python however there can be dangers waiting
    • Resource
    • Using too much
    • Multi-threading
    • Deadlock
      • After you ... no After you
    • Race
      • Some factor alters the order of the flow of processing
    • Interfacing
  • Teamworking

    • Bad Coding Practices
    • Undocumented functions and side-effects
    • Out of date comments leading to mis-understandings.
  • System Design

  • Continious Intergration
  • Local copy of Repo
  • Compile build test
  • Code Coverage

  • Monitoring

    • Dashboards
    • Logs
    • Db Stats
    • SQL
    • Web API
      • Entering
      • Retreiving

Software Methodology

Scrum is characterized by: A living backlog of prioritized work to be done. Completion of a largely fixed set of backlog items in a series of short iterations or sprints.

A brief daily meeting (called a scrum), at which progress is explained, upcoming work is described, and obstacles are raised.

A brief planning session in which the backlog items for the sprint will be defined.

A brief heartbeat retrospective, at which all team members reflect about the past sprint.

Scrum is facilitated by a scrum master, whose primary job is to remove impediments to the ability of the team to deliver the sprint goal. The scrum master is not the leader of the team (as they are self-organizing) but acts as a productivity buffer between the team and any destabilizing influences.

Scrum enables the creation of self-organizing teams by encouraging verbal communication across all team members and across all disciplines that are involved in the project. A key principle of scrum is its recognition that fundamentally empirical challenges cannot be addressed successfully in a traditional “process control” manner. As such, scrum adopts an empirical approach - accepting that the problem cannot be fully understood or defined, focusing instead on maximizing the team's ability to respond in an agile manner to emerging challenges.

CI

We accomplish this using JetBrain product called PyCharm (nosetests, coverage)

  • Jenkins - non-pipeline. Code Coverage, nosetests
  • to a lesser extent. Simple makefiles

Competition

Product

It looks to consist of

  • Market
  • Text
  • Mail
  • Video
  • Messaging

Linux Server, with a dedicated Router.

Competition

Negative about Terminal

  • Price
  • No Touch Screen

Opposition

  • Thompson Reuteurs
  • Money.Net
    • Cheaper
    • Lighter
  • Symphony
    • News
    • Civil

Non Technical

  • Motivation

    • I like technology
    • I like a challenge
  • Communication Style

    • Relaxed
    • clear
    • happy at meeting clients/customers etc

Why Bloomberg ?

Young company, with a very impressive markey share. Which appears to be making commitments to becoming the premiere financial and possibly news data service.

  • Your recruitment process indicates your business is well though-out, and well run.
  • I also want the Asian location.

Questions

  • How will I gain the inside knowledge to fill in the gaps of how your systems interconnect and work. Wiki, e-learning ??
  • does walking up 3 flights of stairs in the office actually increase office productivity ?
  • Who will my line-manager be, and where will they be located ?
  • Are the oppertunities to attend some Data Conferences i.e. PyCon, or Open-Source-Con ??
  • Will I need to learn some Chinese ??

Working

  • hours
  • days (mon-fri)
  • location
  • Dress Code
  • Laptop ?
    • Mine / Company
  • Phone
    • Mine / Company
  • holiday
    • Annual
    • public
  • Pay
    • Basic
    • Housing
    • Overseas/cost of living
    • Other
    • Pension 401k ?
    • Bonus ?
  • Startup
    • Shipping
    • House furnishing
    • Accomodation
    • Agency to use
    • Recommendations
    • visa
    • me
    • wife