HIPAA Cloud thoughts with an architect

Today, while my red team automation was deceptively infiltrating the architect’s pet project, the architect asked me what should one do about keeping the red team automation out of his cloud-enabled PHI-tainted pet project.  Specifically to obtain and maintain a simple baseline.  We started to brainstorm what could a purple teamer efficiently and effectively execute.  Before you know it, we had a list.  Why?  Everyone loves lists of random thoughts.  Isn’t that why Word Clouds are popular?  The format below is not meant to be cohesive and comprehensive.  Just random thoughts translated into bytes on the Internet.  There is a difference between providing legal / security advice vs. providing business advice.  Specifically, if you take these off-the-cuff thoughts and I learn later I am being held responsible because someone relied on that advice to their detriment – no bueno.


No single failure points and defense in depth
Goals – not putting all eggs in one basket.  Instead, execute for redundancy and consensus.  Did you ever bring in cookies for coworkers only to have a random BART rider knock the tin out of your hand, causing the cookies to escape to the far corners of the station?  Well, try not to put all of the cookies into one tin.  Minimize the blast radius.  Consensus is covered below.

In practice
AWS Admin account – one person has the 2FA token.  The other has the password.  One can go further but this is a great start for pet projects.


Lock down Production Access
Add MFA to SSH.  FIDO U2F key is a start.  Arguably require all ssh to be XP / “pair programmed” via key separation

Special laptops for PROD access
Set aside some special machines in the office that are in a locked room.  Only used for SSH access.  Chromebooks work well.  Throw a dropcam / Nest camera in said room to record enter / exit.  Wipe machines on a regular basis.  Disable Intel management engine if possible.

Heavily audit SSH access
Not only setup the above bastion hosts, but restrict who has access and let the stakeholders know when they are accessed via group communications.  Nothing quite like having an executive ask “why were you logged in at 4am last night?  The notification woke up my wife….”  Wake people up when certain commands are issued.  Reputably log every action and key action which goes through the bastion host. OnionID goes a long way.  Storage of logs is just as important but that is a different hairball solved my log management / SIEM efforts.  DR needs to guarantee storage of every action for >10 years.  Immutable logging is a must.

Special rules for who gets production access 

Create a culture where production access is taken seriously.  Obvious controls like background checks.  The real political challenge – stay safe and bond the employees.  $$.  Or make copies of their local and intl. identifications, fingerprint stamps, and anything else needed to issue an intl. arrest warrant in the Philippines. 

Cold Storage of secrets / break glass
Stage 1 – on-premise safe for 24/7 availability.  Separate the passcode and key between parties.   Get a bank deposit box.  The end result being some MFA devices, usb drives, and paper backups in the deposit box(s) and / or safe.

A mature stage – Keys are generated offline in a secure environment.  Split via Shamir’s secret sharing.  Enables redundancy and quorum to restore a key.  Key holders are distributed and follow protocol during key signing ceremonies to verify their identity and assure the integrity of the keys / ceremony.  Hashicorp Vault works well.  Once again, the dropcam / nest cameras are a great detective control.

SIEMs / Log Management
Logging everything which happens across infrastructure.  Storage is cheap.  
Amazing audit trails required.
Deterrent and detective controls.
Need to design for low-latency and high variety logs.  
Probably Kinesis, CloudTrails, Flows, etc…  Too many log sources to mention here.  Large variety of sources, event formats, and event rates at the best of times.  I never heard of anyone stating "Gee, I wish we had less logging."  Correction - when running dtrace or light system tracing on a Production system.



Anomaly Detection
Push warning to group communications.  Typical events are brute-forcing and hitting rate-limiting.

Trigger paging to wake someone up even if it is the middle of the night.  These require immediate attention.  An unusual movement of customer data is a typical event.

Critical Issues
Key phrase: contextual alerting with adaptive responses.  These trigger kill switches that gracefully shuts down relevant systems / services.  These kill switches require their own special ceremony (think manufacturing maintenance ISO9600 efforts to place physical locks on broken machinery) to re-enable.   Typical events include unauthorized access to certain machines / services.
Consensus-based deploys


If one is not adopting this philosophy, most likely the architects are stuck in 90s / early 00s design patterns (The M&M candy security model) or antiquated technologies are in use.  Zero Proof / Trust security models work great and are easy to implement.

Typical workflow – every pull request requires approvals (however that is achieved, +1 via github? .) Each branch or repo may have its’ own sensitivity value (more consensus vs. less based upon the risk.)
If someone in development is infected with malware, increase the consensus numbers across the board.  This protects everyone while allowing rational growth.     

Apply the same rational to deployments.   Immutable docker instances.  Just need a Docker file, docker compose file, and envars? to deploy.  Try to avoid Vagrant.  If Vagrant is required, zero out the empty space in the partitions.  Kubernetes vs. Docker vs. CoreOS vs. <insert technology stolen from VMWare ThinApp> - find what works best for your skillset and future growth.  


Benchmarks go a long way.  See github.com/cloudsriseup for simple scripts to execute to close out the Big 4 (AWS/GCP/Azure/Alibaba) cloud providers gaps.  These scripts cover the benchmark / STIG / “standards” gaps – minus the “Hardware 2FA for AWS Root account” and similar items.  

Make the account(s) / services HIPAA / HITRUST adherent – another story for another time.  SafeNet / Gemalto HSM on-premise backup to AWS’s Gemalto HSM is a pain but will need to test periodically to ensure happiness, unlike Isaac Hall https://techcrunch.com/2012/09/05/recurly-failure/  . 

Strive to obtain HIPAA / HITRUST expert determination.  Close gaps identified by expert determination.  Keep determination in Legal share and court-approved escrow storage.


Red Team drills

If the red team wins, everyone loses.  Gamify while closing out executives’ risk concerns.

You know what you designed.  Red Team tells you what was implemented.


Bounty program

Fresh set of eyes. 

The benefits of 1000 one trick ponies instead of 3 one trick ponies are beyond reproach.


3rd party greybox tests

Afford the best vendor for the appropriate service.  For instance, I wouldn’t let Spider Labs near my systems.  But I would let Bishop Fox take on my mobile applications.


PII / PHI handling vendors

This will become the bane of your professional existence.  AWS will “accelerate maintenance schedules with 2 hour notifications in their commitment to your security.”  In other words, we are busy patching our EC2 infrastructure and our high availability is lacking.  Our lack of planning is now your emergency.

The best S3 bucket data breaches were not with the data owners, but the third party data custodians who would allow ANY AWS IAM identity to manipulate the bucket or flat out allow anonymous access.  Defeating the purpose of the original goals above. 

If the vendor is handling your customers’ PHI, get a sub BAA, not a BA contract from said vendor.  Very different in scope and legal liabilities.


Incident Response

It is a lifestyle, not a process.  Practice makes permament.  Rehearse or perform resilience testing.  Nothing works quite like pulling random cables out of devices.

Everything changes if one has to handle clean vs. white vs. ISO 1 rooms.  See how long it takes for one to introduce a new electronic device into an ISO 1 room.


New engineer / ops onboarding
Enough said.  Pour the koolaid on thick and employ simple, actionable opportunities to change behaviors. Nothing quite like saying welcome by escalating phishing attacks from the CEO while HR is socially engineering the new hire. 

When your SIEM models are not enough

Upon suggestion from Mr. Hay, I took https://sigopt.com/ for a spin.  I plugged it into our SIEM and Vulnerability models.  I am astonished.  Just when I thought every bit of value was squeezed from the systems, it is continuing to pull out indicators and APT actors like candy at a weight loss camp.  One should give it a spin when they need to further optimize their models.  For blackhats, this technique will become a significant pain as additional academic savy private sector practitioners move beyond log management and playbooks. 

First 100 Days

Last paragraph first

Where from Here?

With that being said, the takeaways for your first 100 days; 

  • Maximize success by creating detailed plans for activities for the first months
  • Set priorities carefully and avoid over commitment.  Try to start with the top five pressing issues and select two for the first 2-3 months.
  • Stay away from technical unless it is absolutely required for the role or to earn respect.  Focus on the relationship of security to the business units
  • Significant amounts of your time will be spent in a reactive manner handling unpredictable events or other peoples’ lack of planning is now your emergency 
  • Lastly, it goes without saying to never say anything negative of the predecessor’s implemented roadmaps and actions in front of peers, stakeholders, or team

A friend took up a new executive career path but didn't know how to start.  She reached out to me and ask for my thoughts.  I thought about it and came up with an overly verbose list of things need doing.  But it would take well over 30 minutes to cover each item.  While writing up the answer, I thought to make this public knowledge in the interest of furthering authentic discussions on what is achievable for various security programs in their first 100 days.  The information below is drawn from lessons learned, applied knowledge / experiments, personal notes sold to researchers, and old college courses on organizational theory.  

Below is a straw-man for generic security programs.  I hope you are able to take away a few key points and not repeat my hard lessons learned.  The post is structured to be easily tailored for Blue Team, Red Team, Purple Team, AppSec, OpsSec, IT Sec, PhySec, etc..  As you might imagine, Blue Team is not going to know when to run static source code analysis nor when to provide feedback.  Just as PhySec will only know how to secure physical assets and introduce tamper-evident seals to shredders.  How one performs in their first 100 days is critical to their success or failure.  It is a honeymoon period to formulate a course of action, make connections, establish relationships, and communicate a personal mgt. style.  It is critical to establish yourself and create basic perceptions others will associate with your subsequent plans and actions.  I break down the first 100 days into six phases, each overlapping with recommended durations.  It is expected this model will not fit all organizations.  Chaotic organizations may require lighter, friction-less touches within smaller time periods.  Mature organizations will have their own model and work within the scale of years.


The desired agenda

  • Starts prior to the Hire Date
  • Focus on subset of priorities and drive actions to near-term improves (easy wins)
  • Bridge delivery of business value and internal program excellence
  • Forge respectable relationships with stakeholders (Finance, Legal, Ops, Support, Engineering, Infrastructure, etc.…)
  • Establish a baseline which become the foundation to measure against (if one didn’t exit prior)
  • Communicate the program’s compelling future
  • Highlight future opportunities while encouraging to learn from past mistakes
  • Define and communicate realistic and measurable, time-bound goals, and establish tracking systems to check when goals are achieved
  • Provide your mgr. creditability and elevate the image of the program, department, and organization


The Six Phases

Preparation makes Permanent

                Take key actions to inform yourself.  Learn about your directs / indirects, peers, staff, and draft communications for Day 1.



                Gain comprehensive insight into the current state of the program



                Synthesizes the long assess phase into areas of focus.  Leading to the transformation of everything you learned into a blueprint



                Delivers visible results. Focus on the two key issues identified per the interim program strategy, but seek to address the other foundational areas



                Start providing evidence of your impact.  The overlap with the Execution phase provides the opportunity for feedback so that Execution phase activities and deliverables may be adjusted – ensuring the desired, repeatable, predictable results





Preparation makes Permanent (-15-15)

Before Hire Date


  • An arrangement and understanding of role, expectations of you – among yourself, management, senior stakeholders, and new staff
  • Glean management philosophies and approaches
  • Set understanding of your management philosophies and approaches to directs / indirects


Soft Skills to Brush Up

Communication skills

  • Business language – use technical language as appropriate to the correct audiences
  • Clear, concise and consistent messages (as they are interpreted, not communicated) in forums and across listeners / readers
  • Focus on what is specific to the org.’s performance
  • Connect specific plans to the orgs.’s strategies and investments
  • Socialize plans to peers and leadership with active solicitation
  • Write a 100 word or less short bio.  Neutral messaging, no spin.  Key priorities at work, personal life, values, and integrity.

Business and Technical owner discussions prep

                5 or less questions – open and specific.  Attempt to yield insightful conversation beyond shaking hands “What is your perception and satisfaction level on the current state of X program and organization?”  Then you will be expected to resolve them as quickly as possible according to them.  Typically their priorities, general expectations, and chronic pain / suffering or useless roadblocks



  • Similar questions for directs and indirects as above
  • Key work challenges, constraints, perceptions, and satisfaction within team, department, org., and business unit


On First Hire Date

Key Communication Opportunities

  • Meet and Greet
  • Outcomes – approachable and available.  The walk away - still gathering information and not ready to make decisions or changes.  No opinions offered at this time.
  • Structure
    • Intro prior intro-message drafted in advance.  State when you will report back to the team on updates
    • Team self-introductions in any manner of their choosing and ask any question of their choosing. Nothing off limits
    • Remember a detail about each person
    • Be cognizant of biases (political, generational, social, etc..) which may remain from predecessors
    • No need to come on strong (ensure not to appear so.)  Do not appear as a threat
    • With one-two-ones with directs (and eventual indirects) – what are the concerns, priorities, career aspirations.  Which one understand and can describe the big pictures?  Which are caught in a mud hole or silo?  Where or who needs immediate help?
    • Publish Meet and Greet notes with the wider organization (as appropriate)


Specific, Measurable, Attainable, Relevant, and Timely Actions

  • Logistics – Work with HR and other representatives to setup meet and greets on the first day.  This will help set the tone from day 1 about productivity and expect the same from others
  • Org. Structure – Grab org charts and learn as much lore about the movers, shakers, and levers within the organization and specifically within Finance, Legal, Ops, Engineering, Support, and Security (Info, App, GRC, Ops, SecOps, Red, and Physical.)  Make sure to mark the untouchables
  • Key Players – Work with supervisor to maintain a list of key players to meet with the first week
  • New Connections – Thank you notes to interview team.  Setup a few lunches with said party
  • Program specific action items – Based upon the needs of the business and organization.  Each institution is different as is their programs


Last – Regroup with manager.  Cover key challenges and opportunities from your POV.  Prelim strategic vision.  Future communication schedule, one-two-one expectations, and other manager / report relationship items


Assess (0-30)


  • Insight into current state of X program
  • What is working and not working?
  • Top five challenges which are prioritized for the first 2-3 months


Key Communication Opportunities

                Meet team leads within your program

  • Opinions on state
  • Informed opinions on urgent tasks and coach the leads on how to approach them
  • Solicit input from leads and support them to make it clear you can’t achieve anything alone



                                Grab the preselect top key stakeholder list and meet with three of them

  • Acquire opinions on current program and any changes they might suggest

                                From the key stakeholder list, actively engage with pivotal business leaders.  May need to pull from Executive Security Cell team. 

  • They will understand you will put their business needs first as you craft and execute your plans.  This will help cement crucial relationships.
  • Start off with an open and cooperative relationship.  Grab their objectives and concerns with the X program function.  Ask advice and write down their answers in front of them.  More than likely, they will give you a roadmap to handle their immediate and long term needs

Identify the movers and shakers

  • These influencers will help you avoid being a lone voice trying to initiate cultural change


Specific, Measurable, Attainable, Relevant, and Timely Actions

  • Document findings – succinct report to mgr. and other individuals mgr. may care to collaborate with
  • Artifact gathering – Recent steering committee meetings minutes (IA, Board, Security Cell, Executive, Engineering, Legal, Ops, Finance, Support, etc.…)Grab the last two years’ worth of compliance / GRC artifacts (reports, ERM, Role’s risk artifacts, SSAE SAS70 / SOC 123, Report on Controls, Pentests, Tooling reporting (static source code analysis, dynamic testing – black, white, and grey), Code repository metrics, IP metrics (cyclomatic complexity, ELOC, etc.…), architectural diagrams, Vuln. Mgt. reports, SecOps incidents, etc.…
  • Loosely memorize infosec documentation – policies, assurances, procedures (if you are lucky), mechanisms, charters, principles, strategies, program plans, roadmaps, etc.…
  • Request materials available on the actual expenses and capital spending activities.  If the organization is mature, try to grab forecasting datasets.
  • Program specific action items – Based upon the needs of the business and organization.  Each institution is different as is their programs


Last – Reset? Or set expectations with manager and the role’s authority as it relates with in the institution


Plot (15-45)


  • Planned budget for the next 3-4 months
  • Program vision draft
  • An interim strategy for 6-12 months – which identifies two key issues over the next 2-3 months.  It is expected to change so do not spend much time in this area for chaotic organizations


Key Communication Opportunities

                Program Vision Draft

  • Clear, succinct vision.  Frameworks which fit the culture may be appropriate.  BSIMM, ISO, NIST, NIST CSF, etc.…. will suffice.  They will assist as a planning guide and executive communications.  Draft and share with directs / indirects, all expected stakeholders – solicit their input

                Interim Program Strategy

  • Where do we wish to be?
  • Where are we?
  • Gap Analysis between the top two questions.   The outcome will be a list of current and new projects.



Specific, Measurable, Attainable, Relevant, and Timely Actions

  • Program Vision Draft
  • Interim Program Strategy
  • Grab other departments strategy and budget documents – great insight into the rigor, structure, and expected level of strategic planning within the institution
  • Ensure to grab prior program strategy documents / budgets – Works great to frame discussions on what worked and didn’t work with stakeholders
  • Acquire the related departments’ plotting principles and guidelines to allow you to align to business requirements while planning and plotting


Last – None


Execution (15-45)


  • If applicable, draft program charter
  • Publication of interim program strategy – ensure to include the key issues identified earlier
  • Closer relationships with peers and upper mgt...
  • Initiation of the rest of work required to establish technical and business creditability and foundations for the program (includes budget)


Key Communication Opportunities

                If applicable, Program Charter Draft

  • Establishes formal accountabilities and “executive” mandates.  Try to write it for 3 years.  But realistically will change in a few months due to unforeseen incidents or stakeholder turnover in reactive organizations.  No jargon.  Avoid specific trends or silver bullets.  Simple, succinct phrases “To protect and server,”  “Like water out of a tap,” “track all Production IP deployments,” “Each portfolio application will receive X attention,” etc.… 

                Team and one-two-ones

  • Ask to review their scope and to consider their performance metrics.  Objectives will be clear and scope well-defined.  ASK WHAT YOU CAN DO TO MAKE THEM SUCCESSFUL and follow through!  Work to find practical alternatives when expectations do not meet reality.

                Schedule and conduct monthly team updates

  • The monthly timetable will depend on the culture and rate of change within an organization.  May be monthly.  Could be shorter.  Could be longer.
  • Consistent measurements of the teams.  Standard update report from leads will give everyone the opportunity to glean what their peers are up to and pass that to their reports.  May not be needed for small organizations or chaotic orgs.  Ensure to keep the meeting to less than 21 minutes.  Gives the teams a sense of ownership and pride while increasing their confidence and public speaking skills.

                Quarterly Upper Mgt. Updates

  • Listen to their questions.  Try not to let them wander like Directors will do at a Board meeting or analysts in an individual contributor meeting.
  • Follow a consistent format
    • What did you say you were going to do this period?
    • What did you accomplish?
    • What is the business value in relation to the accomplishments?
    • What business value would the executive team like to see delivered in the next period?



Specific, Measurable, Attainable, Relevant, and Timely Actions

  • Team effectiveness coaching and leadership
    • Give leads their first assignment
    • Define scope and develop metrics.  Emphasize collaboration and plan presentation.  If there is a solid lead peer, have them QA.  If needed, create job descriptions.  Ensure to make it clear strong writing and presentation skills are key for non-IC roles.
    • Identify underperforming personnel and develop skills.  Remember, it isn’t their fault if they fail.  It is your fault for setting them up to fail.  Allowing non-exceptional performance will damage the business objectives and demotivate the team.  Most likely, they need some guidance and direction to find their niche.  Assist in creating skill improvement plans with effective use of the resources and projects available.
  • Get involved in current projects
    • It would be stunning if you didn’t inherit prior programs and projects.  You should have a bit of spare time by now to add value.  Do not attempt to take over a project or undervalue a skill. I am certain we all have been there when we think someone thinks little of us when CC’ing their manager, your manager, etc.…  Two expected outcomes from this – keep focused on the business value and keep executive succinct, smarter / not harder, and effective.  Ensure no one leaves with a “winner” or “looser” thought
  • Program Charter Approval
    • Ensure Upper Mgt. sponsorship and approval for the charter.  Schedule face-to-face meetings and read the non-verbal communications.  It is essential for the program to confirm what Upper Mgt. expects from you.  Also presents an opportunity to establish a close working relationship.
  • Budget
    • Take a look at the next 6-12 months.  Highlight changes in green, yellow, and red.  Look for trends and outliers in the expense categories.  Most likely will be productive for financial savings.  Create a plan for cost reduction so you can put it out of your hat on a rainy day.
    • Write a funding plan for the program’s first iteration.  Mainly will cover transformational work.  It is expected to be over and above the operating budget
  • Governance
    • Evaluate the effectiveness of any governance process – suggested to use the prior assessment as starting point.  You will walk away understanding effective decision making right – linked to accountability, responsibility, and authority
    • Leverage supplemental resources from external providers when internal resources do not exist to drive action
    • Take advantage specific individuals may be more flexible in offering assistance as they seek to influence you.  Beware, you may lose control of your soul in the future
  • Program specific action items – Based upon the needs of the business and organization.  Each institution is different as is their programs


Last – None


Measurement (45-100)


  • Initial status report for Upper Mgt. and Executive Security / IA Committee
  • Evidence of early progress and achievements
  • Foundations of a reporting framework
  • First Quarter status report


Key Communication Opportunities

  • Highly Wins and Successes
    • Schedule meetings with directs / indirects, mgr., and stakeholders to gather their thoughts on progress and challenges.  Collate the findings into a first quarter status report.  Report what is only relevant to the audience(s.)  Interpret the metrics and provide recommended courses of action
  • Monitor program / project success
    • Inherited vs. Initiated projects – Doesn’t matter.  Regular process reports should be brief and focus on the information you need to discuss with the audiences.  Keep TPMs / Project managers focused on telling you how they are doing, not what they are doing.  Ask occasional probing questions at greater levels of detail, to ensure you can articulate the business value of the project team’s efforts


Specific, Measurable, Attainable, Relevant, and Timely Actions

  • See Key Communication Opportunities
  • Program specific action items – Based upon the needs of the business and organization.  Each institution is different as is their programs


Last – None