Speak My Name - Latest Comments

Re: HTTP request debugging in Go with httputil

km — Sun, 12 Dec 2021 23:14:17 -0000

And even after 7 years :)

Re: HTTP request debugging in Go with httputil

morhekil — Fri, 03 Sep 2021 01:08:40 -0000

cookies will be visible in the request headers – they're sent as "Cookie:" http header

Re: HTTP request debugging in Go with httputil

frank — Tue, 31 Aug 2021 05:30:16 -0000

is it possible to verify which cookies are really sent out? They are stored in Client struct and are selected dependent the URL...

Re: Automatic system backups with backup gem and some shell scripting - part 3, large file structures

morhekil — Thu, 16 Jan 2020 17:34:20 -0000

Hi Marcel,

du is going to show you disk space occupied by a directory including all its hard links, so it makes sense that you see all of your backups as having the same size.

You can get a size of linked files separately to compare that, or you can look at all your back directories together - e.g. look here for some good explanations and command examples: https://unix.stackexchange....

Re: Automatic system backups with backup gem and some shell scripting - part 3, large file structures

marcel — Thu, 16 Jan 2020 10:23:54 -0000

Hi there. Thank you for making this post. I focused in on part 3, and use the script in a cronjob to do a backup twice a day, at Noon and 4:00pm. So far, so good.

What I don't get is, that although hard links are used, the target disk is filling up. So far, I haven't been able to find the proper commands to check the actual used disk space. The ones I know of, du -sh for example, or ncdu, show the disk is at 100% capacity. Every backup shows as the same size as the previous backup (1.9T). I know this can't be true, because there are many backups stored on the target disk, which is only a 6TB drive.

And I find it highly unlikely that between the first original (full) backup and - for example - the last one, files have been changed/added so the backups become bigger. A "du -sh" on the original /data drive shows a total usage of 2.0T

Any suggestions as to what might be happening here? Thanks, Marcel

Re: HTTP request debugging in Go with httputil

sonu — Mon, 06 May 2019 01:57:38 -0000

it relevant. the above image was of log file which converts string to json because of jsonformatter

check the following code
dump, err := httputil.DumpRequest(r, true)
if err != nil {
http.Error(w, fmt.Sprint(err), http.StatusInternalServerError)
return
}
logBody := fmt.Sprintf("%q", dump)
fmt.Println(logBody)

when i print logBody to console it give me output as a string not as a json

how to remove this image content from string
https://uploads.disquscdn.c...

Re: HTTP request debugging in Go with httputil

morhekil — Mon, 06 May 2019 01:25:41 -0000

so, this doesn't look relevant to the actual post on this page then, does it? :)

But anyway, if this is a line from your log file - you have two levels of encoding here.

1. JSON container - a JSON object with level and msg keys. Unwrap this first using json.Unmarshal or json.NewDecoder, and get msg value.
2. With msg available (which is, from the looks of it, HTTP request), depending on your needs - you can just cut out the unneeded parts using string substitutions, or parse it with http.ReadRequest ( https://golang.org/pkg/net/... ), and manipulate the resulting http.Request object as you need.

Re: HTTP request debugging in Go with httputil

morhekil — Sun, 05 May 2019 19:12:13 -0000

json.Unmarshal ( https://golang.org/pkg/enco... ) takes a byte array as its input, and allows you to parse it as a json string into a structure you give it - you can take a look at the example there see to how to it can be used.

Re: HTTP request debugging in Go with httputil

morhekil — Sun, 05 May 2019 09:12:26 -0000

can you show an example of your input data and desired output?

Re: Modifying an interface{}-ed struct in Go via reflection

Calvin Cliff — Wed, 16 May 2018 23:40:02 -0000

Thanks for the concise and very useful explanation!

Re: Modifying an interface{}-ed struct in Go via reflection

Alex Arno — Fri, 01 Dec 2017 12:20:09 -0000

Many thanks for the useful article 👍

Re: Automatic system backups with backup gem and some shell scripting - part 2, using backup gem

morhekil — Wed, 20 Sep 2017 20:21:16 -0000

Hi Ornago,

you can use sudoers file to give specific user or group access to run certain commands via sudo without the password. DigitalOcean has a good comprehensive tutorial on how to do it, so this might be a good starting point:

https://www.digitalocean.co...

Re: Automatic system backups with backup gem and some shell scripting - part 2, using backup gem

Ornago — Wed, 20 Sep 2017 16:08:30 -0000

nice tutorial but my backup with "archive.use_sudo" is warning. "sudo: a password is required" how to do that? how can i give the backup gem sudo permissions w/o a password?

Re: HTTP request debugging in Go with httputil

morhekil — Thu, 17 Aug 2017 20:31:22 -0000

that's awesome, I'm glad it has helped you!

Re: HTTP request debugging in Go with httputil

Erl — Thu, 17 Aug 2017 08:53:51 -0000

It helps even after 3 years. Thank you!

Re: Modifying an interface{}-ed struct in Go via reflection

Myles — Wed, 03 May 2017 14:17:48 -0000

Thanks for the nice blog post :)

Re: Simple configuration for Ruby applications

Giulio — Thu, 11 Dec 2014 14:31:33 -0000

thank you, nice solution, as I can see it works very well, except for values like no or false, because of the || in method_missing, solved with

@_settings[name.to_sym].nil?

Re: GZipped JSON files and Logstash

morhekil — Wed, 12 Nov 2014 16:18:17 -0000

Hi Domen,

I'm glad that you found it interested. In its current form this codec expects quite constrained format of incoming data - i.e. names of the files available on the storage, so it won't generalise that well to the case of data being streamed over the wire.

That being said, it should be fairly easy to modify it to work with a stream of data instead - feel free to get in touch if you need any help with this.

Re: GZipped JSON files and Logstash

Domen Kožar — Wed, 12 Nov 2014 10:38:00 -0000

This would be extremely useful to have upstream in logstash. UDP has a max data length limit and compressing messages when hitting that limit would help remove the constraint.

Re: GZipped JSON files and Logstash

morhekil — Thu, 17 Apr 2014 23:45:56 -0000

I can see two possible approaches to this.

First option - you set up another daemon to monitor send file events to redis (could be even a simple bash script, if you don't have too many files), and then the codec setup described in the post will pop them out of the queue and process as usual.

Second option - instead of using redis input, you can write your own input that will monitor the filesystem and send files to processing, as they appear. Current file-based input in logstash (source on github) is using FileWatch gem - you can use that as an example, and write your own input that is only watching for new files, and processes them.

Be careful with FileWatch, though - as far as I can see (based on a cursory reading of its source code), it is just listing files every time it is doing a discovery. If you have a non-trivial number of files (e.g. tens of thousands and up), it will be a slow process, and you'll be much better off ditching it and using kernel events to monitor directory changes - there're gems for that, too.

Re: GZipped JSON files and Logstash

logstashstudent — Thu, 17 Apr 2014 13:49:22 -0000

How would you set this up to pull the name of new .gz files as they are saved to a folder instead of having to enter them manually through the redis cli?

Re: Automatic system backups with backup gem and some shell scripting - part 1, the preparation

morhekil — Fri, 28 Mar 2014 06:19:57 -0000

oh, totally forgot to link it in the post - here it is, enjoy: http://speakmy.name/2014/01...

And fixed the post, thanks for the heads up.

Re: GZipped JSON files and Logstash

databoy — Wed, 26 Mar 2014 17:56:19 -0000

awesome post

Re: Long-running MySQL queries may bring you down in sudden ways

SPun2U — Thu, 28 Mar 2013 00:37:25 -0000

Nice read!

Re: Simple configuration for Ruby applications

Rails Newbie — Wed, 12 Sep 2012 07:22:40 -0000

Thanks, still having trouble with this. Have created the following Gist
git://gist.github.com/37060...

Sorry, I'm still new with working with modules and mixins