AWS S3 IAM user policy setup for bucket

First of all, setup your AWS account – and login into AWS Management Console. Then create any bucket you want.

Next step is to add new IAM user – go to https://console.aws.amazon.com/iam/home?#users, select Users and click Create new user button.

iam_create

Click Create and copy/paste or download your access keys.

iam_access

Then close window and click to summary tab – you’ll need to copy ARN notation of this user

iam_summary

 

s3cmd

To be able to sync some folders and/or use s3cmd commandline tool, you need to setup some more policies. Click Permission tab and then Attach User Policy. Then choose Policy Generator and follow these steps:

manage_user_policySet permissions as shown in following image – we need ListAllMyBuckets for :::* and ListBucket and PutObject for the one, specified bucket.

user_policy_editor

 

Click Continue and save new policy

set_permission

You can always edit the policy using Manage Policy. You should create this kind of policy file:

That’s all. Lets move to bucket policy.

Bucket policy

Now you can open Policy Generator - http://awspolicygen.s3.amazonaws.com/policygen.html and start adding new items.

First of all, select your policy type – S3

policy_select_type

and then fill new policy – we need two policies – one for all users to read objects (GetObject) and second for your new user, to be able to upload, delete and get object as well.

public_policy_form

and click Add Statement. The list below the form should look like

public_policy_list

So, add your user the same way. As principal add your ARN notation (which you copied before) and add Actions as shown below.

policy_user

All set. You need to click Generate Policy button and copy freshly generated JSON policy to your clipboard

Last step is to add this policy to the bucket. Go back to your S3 console (https://console.aws.amazon.com/s3/home), click loupe icon next to bucket name and open Permission section. Click on Edit Bucket Policy and paste the generated policy into that window. Save and you’re done.

bucket_permission

 

You’ll need API keys for use with Paperclip or any other S3 storage engine, so keep them safe.

PS: when you plan to use s3 as storage for your static files, avoid using underscore in your bucket name – as it does not comply with valid FQDN. ‘sample_bucket’ was used only as an example, in the real word you should use ‘sample-bucked’.

ATtiny13 – Hello World! :)

Finally managed to get into AVR programming. Using OSX, which I found as the worst platform for doing any kind of embed programming :( Sad. Anyway, there’s my first ATtiny schematics and code. I’m using AVR Dragon to flash code into the MCU.

tiny13-spi

The circuit is quite simple. I just confirm SPI flashing works and I’m able to turn the LED on. So here’s some code (using avr-gcc to compile and avrdure to upload). Create new project using avr-project and add this to main.c file:

Now we can compile the firmware and upload to our MCU.

No errors. Great. Connect AVR Dragon and upload our new firmware:

tiny13-spi-foto

For those, interested in .hex file – this is how it looks like:

and I made some commented assembler output (base is taken from avr-objdump -S main.elf).

 

Optimizing fluentd

We’re currently using (for one part of our infrastructure) logging into elasticsearch. We have fluentd collectors and kibana interface for viewing and searching through the logs. fluentd This is how it works. Logs are sent to fluentd forwarder and then over the network to fluentd collector, which pushes all the logs to elasticsearch. As we have plenty of logs, we need to incorporate some buffering – on both sides – using buffer_file statement in the fluentd config. Here is a part of our fluentd config from forwarder

and the same for the collector

So. For the forwarder, we’re using buffer with max 4096 8MB chunks = 32GB of buffer space. Forwarder is flushing every 10secs. For collector, we use bigger chunks, as elasticsearch is capable to handle it – but not using default 256MB chunks due to memory limitations. Flushing period is longer – and should be – recommended value is 5minutes. We can keep up to 64Gigs of buffer data.

What happens if one of the fluentd dies. Some data will be probably lost, when unsaved to buffer. But. When there’s connection lost or collector fluentd isn’t running, all logs, collected by forwarder, are stored into the buffer – and sent later. Which is great. The same when ES is down for some reason, collector node is still receiving data and is able to continue sending into ES after full recovery.

PS: don’t forget to make some tweaks to the system itself, like raise the limit for max files opened and some tcp tunning.

Securing kibana + elasticsearch

After some successful setup of Kibana + es for fluentd there’s a need to secure whole website. So I decided to use nginx and basic auth. I assume you have standard configuration – with es running on localhost:9200.

and now modify nginx config:

Duplicity – BackendException: ssh connection to server:22 failed: Unknown server

Booom! After reinstall of one of our servers I got into this. Weird error. It’s caused by paramiko. There’s no code fix available, but reason is simple – and fix too.

Connect to your box, and simply remove two files from /etc/ssh directory

so, remove these two files:

Then clean up ~/.ssh/known_hosts  file on the box your’re running backup from

connect using ssh to backup server from that host (to write id_rsa keys into known_hosts file)

and run duplicate again.

voila! :)

1. Dort – WIP

Korpus I – kakaový

  • 270g hladké mouky (5.10 Kč)
  • 100ml vlažné vody (..)
  • 150ml oleje (6 Kč)
  • 250g cukr moučka (6.50 Kč)
  • 10g prášku do pečiva (5 Kč)
  • 5 vajec (12 Kč)
  • 2-3 lžíce pravého kakaa ( ..)

Bílky vyšlehat na sníh, cukr utřít se žloutky, přidat zbytek. Pečeme při 160oC cca hodinu (3,5kW – cca 10Kč). Aby dort byl plochý, můžeme překlopit vzhůru nohama na podložku.

Čas přípravy – 30min (50Kč), cena za korpus – 44,60 + 50Kč = 94,60 Kč

Korpus II – oříškový

  • lískové ořechy, 100g (20 Kč)
  • vanilkové aroma (lahvička 26 Kč)

ad korpus I – místo kakaa drcené ořechy, pár kapek vanilkového aroma. Cena cca 115 Kč.

Krém I – máslový

  • 500ml mléka (9 Kč)
  • 200g cukru krupice (6 Kč)
  • 60g vanilkového pudinku (většinou 1 a 1/2 sáčku) (2x 16Kč)
  • 1 vejce
  • 2 a 1/4 másla (máslo á 39 Kč)

Máslo nechat povolit, aby mělo pokojovou teplotu. V kastrůlku mléko, pudink, cukr, projít varem. Ve studené lázni vymíchat – dokud to nepálí. Pak postupně zašleháme máslo. Přidal jsem i půl víčka rumu. Dát vychladit do lednice.

Čas: 30minut, cena 135Kč + 50Kč práce – 185 Kč

Krém II – malinové mascarpone

  • 500g mascarpone (2x 40Kč)
  • 200ml 33% smetany (19 Kč)
  • 1 zakysaná smetana (13 Kč)
  • ztužovač šlehačky (8 Kč)
  • 120g cukru (3,12 Kč)
  • 1ks vanilkového cukru (2 Kč)
  • maliny z mrazáku (10 Kč)

Smetanu vyšlehat se ztužovačem, pak vše ostatní smíchat a vymíchat dohladka.

Čas: 30minut, cena 136Kč + 50Kč práce – 186 Kč

Na dort je třeba také potahovací hmota – té bylo čtyřnásobek množství v tomto postu. Cena tedy 4x30Kč + cca 20minut práce – 34Kč – celkem za potahovací hmotu 154Kč

Náklady na dort jsou tedy celkem 735Kč.

 

falešný marcipán – fondant

Můj první pokus – peču dort. Chtěl jsem potahovat, takže jsem v Makru omylem zakoupil Fondánovou hmotu – od Kovandy. Kyblík 4kg – mají i menší balení. Na kyblíku je napsáno, že pokud z toho má být potahovací hmota, je třeba to smíchat se sušeným mlékem, olejem atd.. Takže jsem zkusil internet, našel pár “recepisů” a interpoloval. Výsledek docela překvapil, na to, že to bylo poprvé :-)

fondant

Ingredience:

  • 100g fondánové hmoty (3.75 Kč)
  • 100g moučkového cukru (s co nejmenší protispékavou složkou) (2.60 Kč)
  • 100g plnotučného (asi se dá i polotučného) sušeného mléka (23 Kč)
  • lžičku oleje (možno nahradit rozpuštěným tukem, např. Omega)
  • lžičku vody
  • lžičku medu
  • mandlové aroma (cca 1/2 lžičky) (lahvička 25 Kč)

Hmotu, cukr a mléko jsem dal do misky, v mikrovlnce prohřál chvilku – hmota se částečně rozpustí. Přidal jsem vše ostatní. Je potřeba si připravit misku s vlažnou vodou, na namáčení rukou. Ingredience mícháme – nebojte se, zůstává to dlouho sypké, je treba to mačkat mezi prsty – asi jako když se dělají bramb. knedlíky. Postupně namáčíme ruce do vody – já namáčel asi 3x – tady pozor – voda se nepřidává – přemokřený “marcipán” nejde nastavit. Je třeba hníst do doby, než hmota bude jednolitá – můžeme si pomoci hnětením na desce, podsypáváme škrobem. Následně do pytlíku, dobře uzavřít. Dá se dobře barvit gelovými barvami – pokud třeba ředit, ředíme alkoholem.

Čas: cca 10minut, cena cca 30 Kč + 17 Kč práce – 47 Kč

MCP7940 – RTC with Pi

As I’m building my own IQ house control system, I need to have RTC in my system. So started playing with MCP7940N and i2c interface. I’m using Raspberry Pi for my experiments with i2c/SPI.

Construction is pretty simple, just use MCP7840N Datasheet.

mcp7940n-schema

 

Then you can start checking with Pi. In my case, I’m using RevB, so my bus has number 1 and RTC got 0x6f address.

what next? Check address 0×00, if the onboard oscilator is running

0×00 means NO, it’s NOT running. So, turn it on

and check few times address 0×00

Increments! Great. That means – our oscilator is working = our clocks are working too. Next step is to set current time and date. But wait. How are values stored and read/written? RTC uses BCD encoding for values, that means, eg. number 94 is stored as 9 and 4, both in 4 bits = 9 is stored as 1001 and 4 as 0100, in hexadecimal 0×94. Easy? Eg. day is using 6 bits, upper two for 0123 values, bottom bits for 0-9, so 31th will be stored as 0×31. For more details please read the PDF – RTCC Memory map. So back to setting the date & time:

Fine. But if we’re setting ’00′ as seconds – address 0×00 – why value 0×80? 7th bit = ST – onboard oscilator enabled.

I’ve written simple Python code to read from RTC and print out the value:

Just run by

Voila! :) RTC is up and running. I’d like to check tomorrow (resp. today) morning, if everything is still working correctly, and can create next module for my IQ house control system :)

capistrano3: run gem binary

I need to setup deploy task to run eye. Tried to deal with gem-wrappers, but had no success. As capistrano3 is using non-interactive ssh, without loading user’s environment (.profile, .bashrc etc), then command, which is not in PATH, it’s not workin.

So, after searching and reading capistrano (capistrano/rvm) source, then sshkit source, I got into this simple solution.

It’s not dependent on any other settings nor knowing what, where rvm is installed.

before change in deploy.rb (not working)

after change in deploy.rb

original code from capistrano is

Index? Yes, please

When altering some tables, I got into optimization adding indexes.

I have one table – users

and doing one query:

and slow-log complains

from his you can see it did full table scan (Rows_examined: 389894) which is kinda wrong.

so, let’s explain

Hmm, “type” shows no index has been used and ALL table scanned. WRONG

Let’s add an index – we have two fields, last_active_at and last_logoff, let’s try with last_logoff only

hmm, still using full table scan. Let’s drop and create combined index

and what our explain friend shows now?

Oops. Explain shows it’s using index (bit better than ALL), but still fetches 385646 rows. Still wrong. Let’s switch columns in index.

What explain says now?

Great! and select?

0.20secs improvement.

Explanation? It’s simple. last_active_at is doing range scan against a value and against second column, thus first referenced. If last_logoff is first in combined index, it can’t be used for last_active_at > part of the query.