Peculiar uses for python's 'else' keyword

I’ve been asked by a few people recently to explain the different uses for the else keyword in python.
python, for a reasons I do not understand, decided to overload the else keyword in ways most people never think of.

The spec isn’t too friendly to beginners either. This is a partial piece of the python grammar specification, for symbols that accept the else keyword, as it is read by the parser generator and used to parse Python source files:

if_stmt: 'if' test ':' suite ('elif' test ':' suite)* ['else' ':' suite]

while_stmt: 'while' test ':' suite ['else' ':' suite]

for_stmt: 'for' exprlist 'in' testlist ':' suite ['else' ':' suite]

try_stmt: ('try' ':' suite
((except_clause ':' suite)+
['else' ':' suite]
['finally' ':' suite] |
'finally' ':' suite))

test: or_test ['if' or_test 'else' test] | lambdef

That’s kind of cryptic, right?

This blog post is primarily aimed at beginners, and covers:

The post has no ordering. You can pick-n-choose the ones you’re not familiar with.

Read More

The Mighty Dictionary

One of pythons strongest built-in data type is the dictionary. You can find it everywhere - from a simple key-value store, to a piece of a complex data structure, and all the way down to one of the basic building block of python’s attribute access mechanism.

It’s probably one of the most important data structures in python, and as such, one needs to understand it.

Read More

How does printf really work?

printf is magical. Did you ever stop and ask yourself how it works?

Contrary to most functions, it accepts a variable number of arguments, and somehow transforms them into a formatted string! The GNU code of printf is pretty simple:

printf (const char *format, ...) {
va_list arg;
int done;

va_start (arg, format);
done = vfprintf (stdout, format, arg);
va_end (arg);

return done;
}

if you look closely, it uses a weird ... syntax, performs a couple of va_ calls and one vfprint call.

to understand printf, we first need to understand how va_ works, then move to printf.

If you’re ready for some hard-core c and assembly, start by reading how va_ works!

Read More

May 2017 Major Outage

The blog suffered a major outage today - it was offline for around six hours. It took me around 90 minutes to get it working once I had time to do so.

Why

A few days ago I created a new droplet for a homebrew scraping project I’ve been working on lately.

Today I decided its time to throw it, and pressed the big, red, destroy button. Then I noticed I deleted the wrong droplet and accidentally deleted my blog!

Disaster Recovery Plan

I had daily backups setup already (remember Poor mans daily blog backups?) which were almost up to date.

All I needed to do is to add a few updates to the latest post. Fortunately, every time I post to LinkedIn, they cache my posts at “oded-ninja.cdn.ampproject.org/c/s/oded.ninja/…”

Up until now I kind of hated that cache. Every time I updated a blog post, it took forever to refresh, which was a real pain. Only this time I was actually grateful that cache existed.

Anyway, have you ever heard of the AMP Project?

The AMP Project is an open-source initiative aiming to make the web better for all. The project enables the creation of websites and ads that are consistently fast, beautiful and high-performing across devices and distribution platforms.

Ghost has pre-baked AMP support, Which I’ve set up to automatically redirect mobile clients.

Snapshots

DigitalOcean provide a droplet snapshot service for only 20% of the cost of the virtual server.

The problem is that snapshots wouldn’t help me at this point, because they get deleted with the droplet. Good thing I created that backup service.

Restore

I forked Ghost a few months ago, and instead of testing my changes on production, I put all my configuration in a private git repository.

That proved really useful for local testing, but especially for restoration. I had a habit of checking changes on up-to-date backups, which meant I restored the blog locally every few days.

Plan Execution

  1. Create a new Droplet
  2. Create new Read-only Deploy Keys
  3. Clone the repository, and build it.
  4. Restore the backup from Google Drive
  5. Pause Cloudflare
  6. Check the website is working
  7. Setup Lets Encrypt
  8. Resume Cloudflare
  9. Re-create my Keybase website proof.

Issues I Encountered

  1. I forgot to update the new droplet’s IP at namecheap, my DNS provider.
  2. When checking the website, before turning on SSL, I kept getting redirected to the https endpoint. That’s because it was saved in the browsers HSTS set. Fix? clear my browser’s HSTS settings
  3. Chrome has an internal DNS cache which needed to be refreshed. Fix? flush DNS records and sockets at chrome://net-internals/#dns and chrome://net-internals/#sockets respectively.
  4. I didn’t read the Lets Encrypt setup instructions thoroughly and got blocked for an hour after several failed attempts to set it up. Conclusion? RTFM!
  5. I forgot to update my website proof on Keybase.

Lessons Learned

First of all, DON’T PRESS THE BIG, RED, DESTROY BUTTON BEFORE MAKING SURE YOU ARE REMOVING THE RIGHT DROPLET!

Second, I need to make a few adjustments -

  • Enhance my backup script to include nginx’s configuration as well.
  • Setup a mechanism to perform backup after every update, or at least reduce the interval between backups.
  • Automate recovery steps, or at least document them. When I’m under pressure (or drunk), I forget steps :|

Conspiracy Theory: Intel's AMT Vulnerability & The Ken Thomson Hack

Around two weeks ago Intel announced a critical privilege escalation bug that was laying around its Active Management Technology (AMT) login page for the past seven years. The exploit allows a remote attacker to take control of vulnerable devices with ease.

I’ve read many posts that mock the programmer who introduced it, and the (lacking) testing framework and processes to make sure such things never happen.

But, what if no one made a mistake, and the whole thing is a result of an elaborate hack?

  • How much can you trust software?
  • Have you ever checked the validity of the sources your acquire your software from?
  • Can you trust your own code? Have you ever checked the tooling that compiles or runs it?

In 1984, Ken Thompson, a known figure in the hacker community and one of the authors of UNIX, proposed we can’t. In his remarkable paper, Reflections On Trusting Trust, Ken outlines a hack that many considers the worst hack imaginable: The Ken Thomson Hack.

This blog post is a bit long (but worth it!) and made out of three parts:

  1. The AMT Vulnerability
  2. The Ken Thomson Hack
  3. How 1 & 2 lead to a mega conspiracy

! Disclaimer: The conspiracy theory is completely made up.

Interested? Awesome. Start by reading about the AMT vulnerability.

Read More

Ultimate Guide to Winning a Hackathon

I keep running into people that tell me they’re unqualified to go to Hackathons, because their coding skills aren’t good enough. This post is for everyone who wants to win a Hackathon, and specifically to people who avoid them.

I recently participated and won the biggest Hackathon in Israel. I love Hackathons, and I love winning too. Getting a cash reward is fun, but not as much as winning!

After reading the above, you probably think I’m extremely competitive and cocky, but your’e missing the point - my definition for winning a Hackathon is probably different than yours.

Read More

Nuclear Gandhi & Binary Arithmetic

Nuclear Gandhi is the nickname given to the Indian historical figure Mahatma Gandhi as portrayed in the turn-based strategy video game series Civilization.

A bug in the game caused Gandhi, who is a known pacifist in real life, to turn into a nuclear-obsessed maniac that made India the most hostile civilization in the game.

The cause was a glitch in the artificial intelligence settings for Gandhi’s aggression level: Gandhi started with the lowest level of aggression to reflect his historical legacy of pacifism: 1.

When a player adopted democracy in Civilization, their aggression would be automatically reduced by 2, which means that Gandhi’s aggression level should have gone to -1, but instead the aggression level went all the way up to 255, making him as aggressive as a civilization could possibly be.

Interesting right? but how the heck does -1 become 255?

A bit of math

Don’t worry. I’m not going to dive in too deep. There’s a plethora of blog posts and explanations on how integer arithmetic & representation work.

I’ll explain just enough in order for you to understand what’s going on.

Integer representation

510{5_{10}} in 8-bit binary is000001012{00000101_2}, pretty straight forward.
But what about510{-5_{10}}? How is it implemented? lets draft a possible solution.

First, we need to know the sign of the number. We’ll reserve the most significant bit for the sign, and use the rest as the values. Second, We’ll make sure we don’t break compatibility and set the sign bit for positive numbers to zero, and negative numbers to one. In this scenario a
signed 8-bit number would range from -127 to 127.

Now, in our hacky system,510{5_{10}} won’t change, and510{-5_{10}} will be100001012{10000101_2}.

But here’s the catch - regular arithmetic doesn’t work:

510+510=000001012+100001012=100010102=10100{5_{10}} + {-5_{10}} = {00000101_2} + {10000101_2} = {10001010_2} = {-10_{10}} \ne 0

We can build custom assembly arithmetic, but that’s an over-kill.

Two’s complement

Two’s complement is a mathematical operation on binary numbers, as well as a binary signed number representation based on this operation. Its wide use in computing makes it the most important example of a radix complement. - Wikipedia

TL;DR: a different system that makes arithmetic work as you’d expect.

// 00000101
int x = 5;
// ~x = 11111010
// ~x + 1 = 11111011
int negativeX = ~x + 1;

For example, addition of510{5_{10}} and510{-5_{10}} works like we expect:510+510=000001012+111110112=000000002{5_{10}} + {-5_{10}} = {00000101_2} + {11111011_2} = {00000000_2}

More information is out of scope for this blog post. If you’re interested, start from the answers for What is “2’s Complement”? on StackOverflow.

Ok, so what happened?

A Civilization’s aggression level was saved as an unsigned char, which can’t represent negative values.

Gandhi’s aggression level started at110{1_{10}}, and when democracy arrived, it was reduced by two:110210=000000012000000102=000000012+111111102=111111112{1_{10}} - {2_{10}} = {00000001_2} - {00000010_2} = {00000001_2} {\color{red} +} {11111110_2} = {11111111_2}

if the aggression level variable was signed, then the binary would be interpreted as110{-1_{10}}, which is what we’d expect. Instead, it was unsigned, which means it got interpreted as25510{255_{10}}.

… And Gandhi turned from a pacifist into a warmonger: “Greating from M. Gandhi, ruler and king of the Indians… Our words are backed with NUCLEAR WEAPONS!”

Do you REALLY understand floating points?

I’m going to ask you a couple of questions. If you answer all of them correctly, and understand why - good job! this post is not for you.

Otherwise, If you’re a normal human being, consider reading this post. It’ll save you hours of useless debugging. Honestly, If the engineering who built Ariane V read it (and set their compiler to warning as error) their rocket wouldn’t blow up.

What’s the answer? yes or no?

float x = 0.7;
printf(x == 0.7 ? "yes" : "no")

What will be printed?

float x = 4 / 3;
printf("%f", x);

What’s the answer? yes or no?

float x = 1.0/3.0;
double y = 1.0/1234567.0;
printf(((x+y) - x) == y ? "yes" : "no");

Are both lines equal?

float x = 0.20;
double y = 0.20;

printf("%4.20f\n", x);
printf("%4.20f\n", y);

Now that I’ve got your attention, lets go over the answers real quick. Once you get to the end of this blog post, You’ll understand them fully and be able to impress your coworkers with useless knowledge.

float x = 0.7;
printf(x == 0.7 ? "yes" : "no")

output: no.

float x = 4 / 3;
printf("%.3f", x);

output: 1.000

float x = 1.0/3.0;
double y = 1.0/1234567.0;
printf(((x+y) - x) == y ? "yes" : "no");

output: no.

float x = 0.20;
double y = 0.20;

printf("%4.20f\n", x);
printf("%4.20f\n", y);

output:

0.20000000298023223877
0.20000000000000001110

Read More