[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 07 02:06:44 dist-test-slave-v0bg krb5kdc[32276](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 07 02:06:44 dist-test-slave-v0bg krb5kdc[32276](info): set up 2 sockets
May 07 02:06:44 dist-test-slave-v0bg krb5kdc[32276](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20240507 02:06:45.327476 32269 mini_kdc.cc:121] Time spent starting KDC: real 0.875s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/krb5kdc/test-user.keytab.
May 07 02:06:45 dist-test-slave-v0bg krb5kdc[32276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-07T02:06:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-07T02:06:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.131.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab.
Entry for principal kudu/127.31.131.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab.
I20240507 02:06:45.424451 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:34673
--webserver_interface=127.31.131.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab
--principal=kudu/127.31.131.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:44201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.131.126:34673
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:06:45.500876 32289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:06:45.501055 32289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:06:45.501073 32289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:06:45.502279 32289 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240507 02:06:45.502313 32289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:06:45.502326 32289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:06:45.502336 32289 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240507 02:06:45.502352 32289 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240507 02:06:45.503878 32289 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:44201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.131.126:34673
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab
--principal=kudu/127.31.131.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:34673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.131.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32289
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:06:45.504032 32289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:06:45.504173 32289 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:06:45.507822 32295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:06:45.507907 32289 server_base.cc:1034] running on GCE node
W20240507 02:06:45.508008 32294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:06:45.508148 32297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:06:45.508349 32289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:06:45.508536 32289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:06:45.509665 32289 hybrid_clock.cc:648] HybridClock initialized: now 1715047605509655 us; error 28 us; skew 500 ppm
May 07 02:06:45 dist-test-slave-v0bg krb5kdc[32276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047605, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:06:45.511767 32289 init.cc:377] Logged in from keytab as kudu/127.31.131.126@KRBTEST.COM (short username kudu)
I20240507 02:06:45.512116 32289 webserver.cc:460] Webserver started at http://127.31.131.126:35355/ using document root <none> and password file <none>
I20240507 02:06:45.512267 32289 fs_manager.cc:359] Metadata directory not provided
I20240507 02:06:45.512305 32289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:06:45.512401 32289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:06:45.513290 32289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/minicluster-data/master-0/data/instance:
uuid: "c1247c32cc03462cb6663d2504e3096b"
format_stamp: "Formatted at 2024-05-07 02:06:45 on dist-test-slave-v0bg"
server_key: "d2f427820f4f6f42bbbae06803a1e75b"
server_key_iv: "86c3413c6eaa4d58853f4e7e912ce294"
server_key_version: "encryptionkey@0"
I20240507 02:06:45.513581 32289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/minicluster-data/master-0/wal/instance:
uuid: "c1247c32cc03462cb6663d2504e3096b"
format_stamp: "Formatted at 2024-05-07 02:06:45 on dist-test-slave-v0bg"
server_key: "d2f427820f4f6f42bbbae06803a1e75b"
server_key_iv: "86c3413c6eaa4d58853f4e7e912ce294"
server_key_version: "encryptionkey@0"
I20240507 02:06:45.514614 32289 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240507 02:06:45.515208 32304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:45.515347 32289 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240507 02:06:45.515414 32289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/minicluster-data/master-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/minicluster-data/master-0/wal
uuid: "c1247c32cc03462cb6663d2504e3096b"
format_stamp: "Formatted at 2024-05-07 02:06:45 on dist-test-slave-v0bg"
server_key: "d2f427820f4f6f42bbbae06803a1e75b"
server_key_iv: "86c3413c6eaa4d58853f4e7e912ce294"
server_key_version: "encryptionkey@0"
I20240507 02:06:45.515467 32289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:06:45.535372 32289 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:06:45.537490 32289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:06:45.537588 32289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:06:45.540876 32289 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.126:34673
I20240507 02:06:45.540908 32356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.126:34673 every 8 connection(s)
I20240507 02:06:45.541184 32289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
I20240507 02:06:45.541867 32357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:06:45.543992 32357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1247c32cc03462cb6663d2504e3096b: Bootstrap starting.
I20240507 02:06:45.544492 32357 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1247c32cc03462cb6663d2504e3096b: Neither blocks nor log segments found. Creating new log.
I20240507 02:06:45.544723 32357 log.cc:824] T 00000000000000000000000000000000 P c1247c32cc03462cb6663d2504e3096b: Log is configured to *not* fsync() on all Append() calls
I20240507 02:06:45.545280 32357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1247c32cc03462cb6663d2504e3096b: No bootstrap required, opened a new log
I20240507 02:06:45.546646 32357 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1247c32cc03462cb6663d2504e3096b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1247c32cc03462cb6663d2504e3096b" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 34673 } }
I20240507 02:06:45.546765 32357 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1247c32cc03462cb6663d2504e3096b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:06:45.546782 32357 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1247c32cc03462cb6663d2504e3096b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1247c32cc03462cb6663d2504e3096b, State: Initialized, Role: FOLLOWER
I20240507 02:06:45.546892 32357 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1247c32cc03462cb6663d2504e3096b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1247c32cc03462cb6663d2504e3096b" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 34673 } }
I20240507 02:06:45.546943 32357 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1247c32cc03462cb6663d2504e3096b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 02:06:45.546963 32357 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1247c32cc03462cb6663d2504e3096b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 02:06:45.547001 32357 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1247c32cc03462cb6663d2504e3096b [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:06:45.547461 32357 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1247c32cc03462cb6663d2504e3096b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1247c32cc03462cb6663d2504e3096b" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 34673 } }
I20240507 02:06:45.547564 32357 leader_election.cc:302] T 00000000000000000000000000000000 P c1247c32cc03462cb6663d2504e3096b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1247c32cc03462cb6663d2504e3096b; no voters:
I20240507 02:06:45.547701 32357 leader_election.cc:288] T 00000000000000000000000000000000 P c1247c32cc03462cb6663d2504e3096b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240507 02:06:45.547740 32360 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1247c32cc03462cb6663d2504e3096b [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:06:45.547767 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 32289
I20240507 02:06:45.547854 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/minicluster-data/master-0/wal/instance
I20240507 02:06:45.547881 32360 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1247c32cc03462cb6663d2504e3096b [term 1 LEADER]: Becoming Leader. State: Replica: c1247c32cc03462cb6663d2504e3096b, State: Running, Role: LEADER
I20240507 02:06:45.547950 32360 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1247c32cc03462cb6663d2504e3096b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1247c32cc03462cb6663d2504e3096b" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 34673 } }
I20240507 02:06:45.548249 32357 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1247c32cc03462cb6663d2504e3096b [sys.catalog]: configured and running, proceeding with master startup.
I20240507 02:06:45.548276 32361 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1247c32cc03462cb6663d2504e3096b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1247c32cc03462cb6663d2504e3096b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1247c32cc03462cb6663d2504e3096b" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 34673 } } }
I20240507 02:06:45.548342 32361 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1247c32cc03462cb6663d2504e3096b [sys.catalog]: This master's current role is: LEADER
I20240507 02:06:45.548461 32362 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1247c32cc03462cb6663d2504e3096b [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1247c32cc03462cb6663d2504e3096b. Latest consensus state: current_term: 1 leader_uuid: "c1247c32cc03462cb6663d2504e3096b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1247c32cc03462cb6663d2504e3096b" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 34673 } } }
I20240507 02:06:45.548499 32362 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1247c32cc03462cb6663d2504e3096b [sys.catalog]: This master's current role is: LEADER
I20240507 02:06:45.548527 32269 external_mini_cluster.cc:1428] Setting key f8de0da8256545689190ca42298bcd71
I20240507 02:06:45.548581 32364 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 02:06:45.549019 32364 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240507 02:06:45.550848 32364 catalog_manager.cc:1347] Generated new cluster ID: 92df737db3514041b32653e230a3f667
I20240507 02:06:45.550902 32364 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
May 07 02:06:45 dist-test-slave-v0bg krb5kdc[32276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:06:45.563521 32373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:45.549166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58676 (local address 127.31.131.126:34673)
0507 02:06:45.549307 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:45.549319 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:45.549810 (+ 491us) server_negotiation.cc:407] Connection header received
0507 02:06:45.555343 (+ 5533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:45.555347 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:45.555383 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:45.555489 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:06:45.556613 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:45.557322 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:45.557875 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:45.558096 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:45.560645 (+ 2549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:06:45.560650 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:06:45.560650 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:06:45.560664 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:06:45.562115 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:45.562448 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:45.562450 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:45.562451 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:45.562492 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:45.562679 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:45.562680 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:45.562680 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:45.562862 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:06:45.562981 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:45.563213 (+ 232us) server_negotiation.cc:299] Negotiation successful
0507 02:06:45.563324 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":58,"threads_started":1}
I20240507 02:06:45.586156 32364 catalog_manager.cc:1370] Generated new certificate authority record
I20240507 02:06:45.586905 32364 catalog_manager.cc:1504] Loading token signing keys...
I20240507 02:06:45.592999 32364 catalog_manager.cc:5857] T 00000000000000000000000000000000 P c1247c32cc03462cb6663d2504e3096b: Generated new TSK 0
I20240507 02:06:45.593165 32364 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.131.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab.
Entry for principal kudu/127.31.131.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab.
I20240507 02:06:45.616492 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.65:0
--local_ip_for_outbound_sockets=127.31.131.65
--webserver_interface=127.31.131.65
--webserver_port=0
--tserver_master_addrs=127.31.131.126:34673
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab
--principal=kudu/127.31.131.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:44201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:06:45.691677 32383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:06:45.691855 32383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:06:45.691885 32383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:06:45.693125 32383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:06:45.693166 32383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:06:45.693208 32383 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.65
I20240507 02:06:45.694759 32383 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:44201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab
--principal=kudu/127.31.131.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.131.65
--webserver_port=0
--tserver_master_addrs=127.31.131.126:34673
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32383
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.65
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:06:45.694957 32383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:06:45.695118 32383 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:06:45.698658 32388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:06:45.698658 32389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:06:45.698844 32383 server_base.cc:1034] running on GCE node
W20240507 02:06:45.699000 32391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:06:45.699177 32383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:06:45.699368 32383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:06:45.700493 32383 hybrid_clock.cc:648] HybridClock initialized: now 1715047605700482 us; error 29 us; skew 500 ppm
May 07 02:06:45 dist-test-slave-v0bg krb5kdc[32276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047605, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:06:45.702652 32383 init.cc:377] Logged in from keytab as kudu/127.31.131.65@KRBTEST.COM (short username kudu)
I20240507 02:06:45.703030 32383 webserver.cc:460] Webserver started at http://127.31.131.65:45095/ using document root <none> and password file <none>
I20240507 02:06:45.703181 32383 fs_manager.cc:359] Metadata directory not provided
I20240507 02:06:45.703215 32383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:06:45.703305 32383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:06:45.704208 32383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/minicluster-data/ts-0/data/instance:
uuid: "34e5aa9442084d7588394816f912e5db"
format_stamp: "Formatted at 2024-05-07 02:06:45 on dist-test-slave-v0bg"
server_key: "65f7cc5d847f76b8c69a867118f3f01f"
server_key_iv: "4d0d749d98c7c03874fdfc3d0fae889a"
server_key_version: "encryptionkey@0"
I20240507 02:06:45.704476 32383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/minicluster-data/ts-0/wal/instance:
uuid: "34e5aa9442084d7588394816f912e5db"
format_stamp: "Formatted at 2024-05-07 02:06:45 on dist-test-slave-v0bg"
server_key: "65f7cc5d847f76b8c69a867118f3f01f"
server_key_iv: "4d0d749d98c7c03874fdfc3d0fae889a"
server_key_version: "encryptionkey@0"
I20240507 02:06:45.705528 32383 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240507 02:06:45.706153 32398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:45.706332 32383 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240507 02:06:45.706401 32383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/minicluster-data/ts-0/wal
uuid: "34e5aa9442084d7588394816f912e5db"
format_stamp: "Formatted at 2024-05-07 02:06:45 on dist-test-slave-v0bg"
server_key: "65f7cc5d847f76b8c69a867118f3f01f"
server_key_iv: "4d0d749d98c7c03874fdfc3d0fae889a"
server_key_version: "encryptionkey@0"
I20240507 02:06:45.706466 32383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:06:45.718884 32383 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:06:45.720369 32383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:06:45.720468 32383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:06:45.720628 32383 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:06:45.720892 32383 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:06:45.720922 32383 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:45.720948 32383 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:06:45.720976 32383 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:45.725924 32383 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.65:38643
I20240507 02:06:45.725979 32511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.65:38643 every 8 connection(s)
I20240507 02:06:45.726218 32383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
I20240507 02:06:45.730213 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 32383
I20240507 02:06:45.730285 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/minicluster-data/ts-0/wal/instance
I20240507 02:06:45.730443 32269 external_mini_cluster.cc:1428] Setting key 4fdde677ae555c92ecb0ac5b32d9da35
May 07 02:06:45 dist-test-slave-v0bg krb5kdc[32276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047605, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.65@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:06:45.733697 32373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:45.726828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:45889 (local address 127.31.131.126:34673)
0507 02:06:45.726873 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:45.726875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:45.727038 (+ 163us) server_negotiation.cc:407] Connection header received
0507 02:06:45.727488 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:45.727488 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:45.727497 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:45.727543 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:06:45.728437 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:45.728925 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:45.729392 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:45.729494 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:45.731386 (+ 1892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:06:45.731389 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:06:45.731389 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:06:45.731395 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:06:45.732570 (+ 1175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:45.732948 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:45.732948 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:45.732949 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:45.732965 (+ 16us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:45.733137 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:45.733137 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:45.733138 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:45.733221 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:06:45.733274 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:45.733505 (+ 231us) server_negotiation.cc:299] Negotiation successful
0507 02:06:45.733589 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20240507 02:06:45.733764 32514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:45.726815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:34673 (local address 127.31.131.65:45889)
0507 02:06:45.726943 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0507 02:06:45.726966 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0507 02:06:45.727411 (+ 445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:06:45.727585 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:06:45.727588 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:06:45.727837 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:06:45.728371 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:45.728376 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:45.728985 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:06:45.728986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:06:45.729344 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:45.729345 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:45.729430 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:45.729829 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:06:45.729840 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:06:45.731297 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:06:45.732642 (+ 1345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:06:45.732648 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:06:45.732650 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:06:45.732892 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:06:45.733027 (+ 135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:06:45.733028 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:06:45.733028 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:06:45.733085 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:06:45.733272 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:06:45.733276 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:06:45.733392 (+ 116us) client_negotiation.cc:769] Sending connection context
0507 02:06:45.733479 (+ 87us) client_negotiation.cc:240] Negotiation successful
0507 02:06:45.733599 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":66,"threads_started":1}
I20240507 02:06:45.734485 32512 heartbeater.cc:347] Connected to a master server at 127.31.131.126:34673
I20240507 02:06:45.734593 32512 heartbeater.cc:464] Registering TS with master...
I20240507 02:06:45.734779 32512 heartbeater.cc:511] Master 127.31.131.126:34673 requested a full tablet report, sending...
I20240507 02:06:45.735241 32321 ts_manager.cc:194] Registered new tserver with Master: 34e5aa9442084d7588394816f912e5db (127.31.131.65:38643)
I20240507 02:06:45.735834 32321 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.131.65@KRBTEST.COM'} at 127.31.131.65:45889
WARNING: no policy specified for kudu/127.31.131.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab.
Entry for principal kudu/127.31.131.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab.
I20240507 02:06:45.750981 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.66:0
--local_ip_for_outbound_sockets=127.31.131.66
--webserver_interface=127.31.131.66
--webserver_port=0
--tserver_master_addrs=127.31.131.126:34673
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab
--principal=kudu/127.31.131.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:44201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:06:45.824970 32517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:06:45.825141 32517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:06:45.825162 32517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:06:45.826377 32517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:06:45.826411 32517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:06:45.826434 32517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.66
I20240507 02:06:45.827879 32517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:44201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab
--principal=kudu/127.31.131.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.131.66
--webserver_port=0
--tserver_master_addrs=127.31.131.126:34673
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.66
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:06:45.828032 32517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:06:45.828166 32517 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:06:45.831673 32523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:06:45.831748 32522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:06:45.831842 32525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:06:45.831784 32517 server_base.cc:1034] running on GCE node
I20240507 02:06:45.832015 32517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:06:45.832206 32517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:06:45.833341 32517 hybrid_clock.cc:648] HybridClock initialized: now 1715047605833326 us; error 27 us; skew 500 ppm
May 07 02:06:45 dist-test-slave-v0bg krb5kdc[32276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047605, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:06:45.835484 32517 init.cc:377] Logged in from keytab as kudu/127.31.131.66@KRBTEST.COM (short username kudu)
I20240507 02:06:45.835860 32517 webserver.cc:460] Webserver started at http://127.31.131.66:38975/ using document root <none> and password file <none>
I20240507 02:06:45.835996 32517 fs_manager.cc:359] Metadata directory not provided
I20240507 02:06:45.836030 32517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:06:45.836102 32517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:06:45.836956 32517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/minicluster-data/ts-1/data/instance:
uuid: "5746820650a64b2a841e03055bc5ff44"
format_stamp: "Formatted at 2024-05-07 02:06:45 on dist-test-slave-v0bg"
server_key: "e29dced6be6b5805fa82f64a3e0c64a1"
server_key_iv: "f148fd249af6afe039dd083bd1fba200"
server_key_version: "encryptionkey@0"
I20240507 02:06:45.837210 32517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/minicluster-data/ts-1/wal/instance:
uuid: "5746820650a64b2a841e03055bc5ff44"
format_stamp: "Formatted at 2024-05-07 02:06:45 on dist-test-slave-v0bg"
server_key: "e29dced6be6b5805fa82f64a3e0c64a1"
server_key_iv: "f148fd249af6afe039dd083bd1fba200"
server_key_version: "encryptionkey@0"
I20240507 02:06:45.838197 32517 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240507 02:06:45.838760 32532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:45.838892 32517 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240507 02:06:45.838959 32517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/minicluster-data/ts-1/wal
uuid: "5746820650a64b2a841e03055bc5ff44"
format_stamp: "Formatted at 2024-05-07 02:06:45 on dist-test-slave-v0bg"
server_key: "e29dced6be6b5805fa82f64a3e0c64a1"
server_key_iv: "f148fd249af6afe039dd083bd1fba200"
server_key_version: "encryptionkey@0"
I20240507 02:06:45.839006 32517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:06:45.870177 32517 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:06:45.871742 32517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:06:45.871831 32517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:06:45.872009 32517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:06:45.872291 32517 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:06:45.872323 32517 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:45.872344 32517 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:06:45.872366 32517 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:45.877147 32517 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.66:35647
I20240507 02:06:45.877194 32645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.66:35647 every 8 connection(s)
I20240507 02:06:45.877457 32517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
May 07 02:06:45 dist-test-slave-v0bg krb5kdc[32276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047605, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.66@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:06:45.884451 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 32517
I20240507 02:06:45.884536 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/minicluster-data/ts-1/wal/instance
I20240507 02:06:45.884701 32269 external_mini_cluster.cc:1428] Setting key c8b7e4fc9441722fd0a8dc6014264e8b
I20240507 02:06:45.884817 32373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:45.878057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.66:51029 (local address 127.31.131.126:34673)
0507 02:06:45.878098 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:45.878100 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:45.878266 (+ 166us) server_negotiation.cc:407] Connection header received
0507 02:06:45.878756 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:45.878756 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:45.878765 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:45.878817 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:06:45.879730 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:45.880205 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:45.880672 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:45.880773 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:45.882525 (+ 1752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:06:45.882530 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:06:45.882530 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:06:45.882538 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:06:45.883788 (+ 1250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:45.884068 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:45.884068 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:45.884069 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:45.884084 (+ 15us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:45.884212 (+ 128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:45.884213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:45.884213 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:45.884290 (+ 77us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:06:45.884332 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:45.884658 (+ 326us) server_negotiation.cc:299] Negotiation successful
0507 02:06:45.884731 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20240507 02:06:45.884918 32648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:45.878032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:34673 (local address 127.31.131.66:51029)
0507 02:06:45.878177 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0507 02:06:45.878206 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0507 02:06:45.878664 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:06:45.878875 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:06:45.878879 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:06:45.879152 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:06:45.879652 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:45.879656 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:45.880264 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:06:45.880265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:06:45.880620 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:45.880621 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:45.880721 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:45.881137 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:06:45.881150 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:06:45.882424 (+ 1274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:06:45.883854 (+ 1430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:06:45.883857 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:06:45.883859 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:06:45.884021 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:06:45.884124 (+ 103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:06:45.884125 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:06:45.884125 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:06:45.884169 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:06:45.884336 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:06:45.884339 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:06:45.884476 (+ 137us) client_negotiation.cc:769] Sending connection context
0507 02:06:45.884629 (+ 153us) client_negotiation.cc:240] Negotiation successful
0507 02:06:45.884757 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":68,"threads_started":1}
I20240507 02:06:45.885434 32646 heartbeater.cc:347] Connected to a master server at 127.31.131.126:34673
I20240507 02:06:45.885530 32646 heartbeater.cc:464] Registering TS with master...
I20240507 02:06:45.885725 32646 heartbeater.cc:511] Master 127.31.131.126:34673 requested a full tablet report, sending...
I20240507 02:06:45.886094 32320 ts_manager.cc:194] Registered new tserver with Master: 5746820650a64b2a841e03055bc5ff44 (127.31.131.66:35647)
I20240507 02:06:45.886456 32320 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.131.66@KRBTEST.COM'} at 127.31.131.66:51029
WARNING: no policy specified for kudu/127.31.131.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab.
Entry for principal kudu/127.31.131.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab.
I20240507 02:06:45.904675 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.67:0
--local_ip_for_outbound_sockets=127.31.131.67
--webserver_interface=127.31.131.67
--webserver_port=0
--tserver_master_addrs=127.31.131.126:34673
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab
--principal=kudu/127.31.131.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:44201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:06:45.978758 32651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:06:45.978928 32651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:06:45.978950 32651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:06:45.980195 32651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:06:45.980230 32651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:06:45.980252 32651 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.67
I20240507 02:06:45.981688 32651 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:44201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab
--principal=kudu/127.31.131.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.131.67
--webserver_port=0
--tserver_master_addrs=127.31.131.126:34673
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32651
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.67
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:06:45.981848 32651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:06:45.981992 32651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:06:45.985525 32659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:06:45.985538 32657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:06:45.985683 32656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:06:45.985620 32651 server_base.cc:1034] running on GCE node
I20240507 02:06:45.985868 32651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:06:45.986058 32651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:06:45.987195 32651 hybrid_clock.cc:648] HybridClock initialized: now 1715047605987180 us; error 27 us; skew 500 ppm
May 07 02:06:45 dist-test-slave-v0bg krb5kdc[32276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047605, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:06:45.989396 32651 init.cc:377] Logged in from keytab as kudu/127.31.131.67@KRBTEST.COM (short username kudu)
I20240507 02:06:45.989738 32651 webserver.cc:460] Webserver started at http://127.31.131.67:34827/ using document root <none> and password file <none>
I20240507 02:06:45.989872 32651 fs_manager.cc:359] Metadata directory not provided
I20240507 02:06:45.989904 32651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:06:45.989980 32651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:06:45.990896 32651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/minicluster-data/ts-2/data/instance:
uuid: "c378dc4e9e36417890aac550ee427789"
format_stamp: "Formatted at 2024-05-07 02:06:45 on dist-test-slave-v0bg"
server_key: "3a694855f71dee1829a6250b680d6c77"
server_key_iv: "dc0274f71ef5bf316347e76dadd1f256"
server_key_version: "encryptionkey@0"
I20240507 02:06:45.991147 32651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/minicluster-data/ts-2/wal/instance:
uuid: "c378dc4e9e36417890aac550ee427789"
format_stamp: "Formatted at 2024-05-07 02:06:45 on dist-test-slave-v0bg"
server_key: "3a694855f71dee1829a6250b680d6c77"
server_key_iv: "dc0274f71ef5bf316347e76dadd1f256"
server_key_version: "encryptionkey@0"
I20240507 02:06:45.992178 32651 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240507 02:06:45.992731 32666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:45.992888 32651 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240507 02:06:45.992952 32651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/minicluster-data/ts-2/wal
uuid: "c378dc4e9e36417890aac550ee427789"
format_stamp: "Formatted at 2024-05-07 02:06:45 on dist-test-slave-v0bg"
server_key: "3a694855f71dee1829a6250b680d6c77"
server_key_iv: "dc0274f71ef5bf316347e76dadd1f256"
server_key_version: "encryptionkey@0"
I20240507 02:06:45.993000 32651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:06:46.005165 32651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:06:46.006896 32651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:06:46.006979 32651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:06:46.007122 32651 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:06:46.007337 32651 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:06:46.007364 32651 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:46.007381 32651 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:06:46.007397 32651 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:46.012157 32651 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.67:40519
I20240507 02:06:46.012190 311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.67:40519 every 8 connection(s)
I20240507 02:06:46.012423 32651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
May 07 02:06:46 dist-test-slave-v0bg krb5kdc[32276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047605, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.67@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:06:46.017951 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 32651
I20240507 02:06:46.018046 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1715047604443657-32269-0/minicluster-data/ts-2/wal/instance
I20240507 02:06:46.018206 32269 external_mini_cluster.cc:1428] Setting key 1043627fdd37c432038c0f214227465d
I20240507 02:06:46.019753 32373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:46.013054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.67:41173 (local address 127.31.131.126:34673)
0507 02:06:46.013110 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:46.013112 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:46.013244 (+ 132us) server_negotiation.cc:407] Connection header received
0507 02:06:46.013688 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:46.013689 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:46.013697 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:46.013744 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:06:46.014589 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:46.015069 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:46.015546 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:46.015658 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:46.017286 (+ 1628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:06:46.017290 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:06:46.017291 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:06:46.017297 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:06:46.018722 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:46.019084 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:46.019085 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:46.019085 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:46.019134 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:46.019314 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:46.019314 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:46.019315 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:46.019401 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:06:46.019449 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:46.019622 (+ 173us) server_negotiation.cc:299] Negotiation successful
0507 02:06:46.019691 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20240507 02:06:46.019824 314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:46.013018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:34673 (local address 127.31.131.67:41173)
0507 02:06:46.013147 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0507 02:06:46.013172 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0507 02:06:46.013613 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:06:46.013791 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:06:46.013795 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:06:46.014040 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:06:46.014516 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:46.014521 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:46.015139 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:06:46.015139 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:06:46.015494 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:46.015495 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:46.015580 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:46.015987 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:06:46.016000 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:06:46.017213 (+ 1213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:06:46.018787 (+ 1574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:06:46.018792 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:06:46.018794 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:06:46.019028 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:06:46.019187 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:06:46.019188 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:06:46.019188 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:06:46.019259 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:06:46.019438 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:06:46.019441 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:06:46.019528 (+ 87us) client_negotiation.cc:769] Sending connection context
0507 02:06:46.019605 (+ 77us) client_negotiation.cc:240] Negotiation successful
0507 02:06:46.019694 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":65,"threads_started":1}
I20240507 02:06:46.020157 312 heartbeater.cc:347] Connected to a master server at 127.31.131.126:34673
I20240507 02:06:46.020233 312 heartbeater.cc:464] Registering TS with master...
I20240507 02:06:46.020406 312 heartbeater.cc:511] Master 127.31.131.126:34673 requested a full tablet report, sending...
I20240507 02:06:46.020749 32320 ts_manager.cc:194] Registered new tserver with Master: c378dc4e9e36417890aac550ee427789 (127.31.131.67:40519)
I20240507 02:06:46.021118 32320 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.131.67@KRBTEST.COM'} at 127.31.131.67:41173
I20240507 02:06:46.029919 32269 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
May 07 02:06:46 dist-test-slave-v0bg krb5kdc[32276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047606, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
May 07 02:06:46 dist-test-slave-v0bg krb5kdc[32276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047606, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.131.65@KRBTEST.COM
I20240507 02:06:46.050880 319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:46.044438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44128 (local address 127.31.131.65:38643)
0507 02:06:46.044576 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:46.044582 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:46.044592 (+ 10us) server_negotiation.cc:407] Connection header received
0507 02:06:46.044616 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:46.044618 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:46.044629 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:46.044693 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:06:46.045275 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:46.045960 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:46.046451 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:46.046645 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:46.048365 (+ 1720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:06:46.048374 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:06:46.048374 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:06:46.048380 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:06:46.049948 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:46.050213 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:46.050216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:46.050217 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:46.050247 (+ 30us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:46.050383 (+ 136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:46.050384 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:46.050384 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:46.050468 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:06:46.050607 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:46.050701 (+ 94us) server_negotiation.cc:299] Negotiation successful
0507 02:06:46.050778 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":68,"threads_started":1}
W20240507 02:06:46.051180 32426 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44128
May 07 02:06:46 dist-test-slave-v0bg krb5kdc[32276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
May 07 02:06:46 dist-test-slave-v0bg krb5kdc[32276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.131.65@KRBTEST.COM
I20240507 02:06:46.069101 319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:46.063735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44136 (local address 127.31.131.65:38643)
0507 02:06:46.063782 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:46.063784 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:46.063794 (+ 10us) server_negotiation.cc:407] Connection header received
0507 02:06:46.063854 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:46.063855 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:46.063864 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:46.063906 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:06:46.064427 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:46.064898 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:46.065381 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:46.065486 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:46.067171 (+ 1685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:06:46.067174 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:06:46.067174 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:06:46.067180 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:06:46.068312 (+ 1132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:46.068563 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:46.068564 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:46.068564 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:46.068583 (+ 19us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:46.068703 (+ 120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:46.068704 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:46.068704 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:46.068768 (+ 64us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:06:46.068845 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:46.068938 (+ 93us) server_negotiation.cc:299] Negotiation successful
0507 02:06:46.069010 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20240507 02:06:46.069520 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 32383
I20240507 02:06:46.073550 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 32517
I20240507 02:06:46.077931 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 32651
I20240507 02:06:46.081661 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 32289
2024-05-07T02:06:46Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1651 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 07 02:06:46 dist-test-slave-v0bg krb5kdc[326](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 07 02:06:46 dist-test-slave-v0bg krb5kdc[326](info): set up 2 sockets
May 07 02:06:46 dist-test-slave-v0bg krb5kdc[326](info): commencing operation
krb5kdc: starting...
W20240507 02:06:46.963379 32269 mini_kdc.cc:121] Time spent starting KDC: real 0.863s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/krb5kdc/test-user.keytab.
May 07 02:06:47 dist-test-slave-v0bg krb5kdc[326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047607, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-07T02:06:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-07T02:06:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.131.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab.
Entry for principal kudu/127.31.131.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab.
I20240507 02:06:47.062402 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:40483
--webserver_interface=127.31.131.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab
--principal=kudu/127.31.131.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:37469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.131.126:40483
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:06:47.135751 339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:06:47.135922 339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:06:47.135941 339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:06:47.137236 339 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240507 02:06:47.137275 339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:06:47.137287 339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:06:47.137310 339 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240507 02:06:47.137332 339 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240507 02:06:47.138839 339 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:37469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.131.126:40483
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab
--principal=kudu/127.31.131.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:40483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.131.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.339
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:06:47.139022 339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:06:47.139192 339 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240507 02:06:47.142725 339 server_base.cc:1034] running on GCE node
W20240507 02:06:47.142721 347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:06:47.142764 344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:06:47.142764 345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:06:47.143057 339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:06:47.143268 339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:06:47.144398 339 hybrid_clock.cc:648] HybridClock initialized: now 1715047607144386 us; error 25 us; skew 500 ppm
May 07 02:06:47 dist-test-slave-v0bg krb5kdc[326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047607, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:06:47.146566 339 init.cc:377] Logged in from keytab as kudu/127.31.131.126@KRBTEST.COM (short username kudu)
I20240507 02:06:47.146910 339 webserver.cc:460] Webserver started at http://127.31.131.126:37615/ using document root <none> and password file <none>
I20240507 02:06:47.147047 339 fs_manager.cc:359] Metadata directory not provided
I20240507 02:06:47.147079 339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:06:47.147156 339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:06:47.148023 339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/minicluster-data/master-0/data/instance:
uuid: "c8d74ba3caad418e85f936e3ad0e6c7e"
format_stamp: "Formatted at 2024-05-07 02:06:47 on dist-test-slave-v0bg"
server_key: "7581c475763ea38103f476820284059a"
server_key_iv: "53bf015a4c7679acc197db077a074c93"
server_key_version: "encryptionkey@0"
I20240507 02:06:47.148276 339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/minicluster-data/master-0/wal/instance:
uuid: "c8d74ba3caad418e85f936e3ad0e6c7e"
format_stamp: "Formatted at 2024-05-07 02:06:47 on dist-test-slave-v0bg"
server_key: "7581c475763ea38103f476820284059a"
server_key_iv: "53bf015a4c7679acc197db077a074c93"
server_key_version: "encryptionkey@0"
I20240507 02:06:47.149276 339 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240507 02:06:47.149814 354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:47.149967 339 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240507 02:06:47.150035 339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/minicluster-data/master-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/minicluster-data/master-0/wal
uuid: "c8d74ba3caad418e85f936e3ad0e6c7e"
format_stamp: "Formatted at 2024-05-07 02:06:47 on dist-test-slave-v0bg"
server_key: "7581c475763ea38103f476820284059a"
server_key_iv: "53bf015a4c7679acc197db077a074c93"
server_key_version: "encryptionkey@0"
I20240507 02:06:47.150082 339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:06:47.157042 339 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:06:47.158519 339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:06:47.158653 339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:06:47.161867 339 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.126:40483
I20240507 02:06:47.161926 406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.126:40483 every 8 connection(s)
I20240507 02:06:47.162158 339 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
I20240507 02:06:47.162918 407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:06:47.164798 407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8d74ba3caad418e85f936e3ad0e6c7e: Bootstrap starting.
I20240507 02:06:47.165230 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 339
I20240507 02:06:47.165302 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/minicluster-data/master-0/wal/instance
I20240507 02:06:47.165295 407 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8d74ba3caad418e85f936e3ad0e6c7e: Neither blocks nor log segments found. Creating new log.
I20240507 02:06:47.165428 32269 external_mini_cluster.cc:1428] Setting key 5fabee5f5c1489ab29de5ca828ae2fb0
I20240507 02:06:47.165522 407 log.cc:824] T 00000000000000000000000000000000 P c8d74ba3caad418e85f936e3ad0e6c7e: Log is configured to *not* fsync() on all Append() calls
I20240507 02:06:47.166200 407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8d74ba3caad418e85f936e3ad0e6c7e: No bootstrap required, opened a new log
I20240507 02:06:47.168169 407 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8d74ba3caad418e85f936e3ad0e6c7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8d74ba3caad418e85f936e3ad0e6c7e" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 40483 } }
I20240507 02:06:47.168282 407 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8d74ba3caad418e85f936e3ad0e6c7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:06:47.168304 407 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8d74ba3caad418e85f936e3ad0e6c7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d74ba3caad418e85f936e3ad0e6c7e, State: Initialized, Role: FOLLOWER
I20240507 02:06:47.168396 407 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8d74ba3caad418e85f936e3ad0e6c7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8d74ba3caad418e85f936e3ad0e6c7e" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 40483 } }
I20240507 02:06:47.168455 407 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8d74ba3caad418e85f936e3ad0e6c7e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 02:06:47.168483 407 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8d74ba3caad418e85f936e3ad0e6c7e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 02:06:47.168525 407 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8d74ba3caad418e85f936e3ad0e6c7e [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:06:47.168979 407 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8d74ba3caad418e85f936e3ad0e6c7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8d74ba3caad418e85f936e3ad0e6c7e" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 40483 } }
I20240507 02:06:47.169080 407 leader_election.cc:302] T 00000000000000000000000000000000 P c8d74ba3caad418e85f936e3ad0e6c7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8d74ba3caad418e85f936e3ad0e6c7e; no voters:
I20240507 02:06:47.169222 407 leader_election.cc:288] T 00000000000000000000000000000000 P c8d74ba3caad418e85f936e3ad0e6c7e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240507 02:06:47.169275 412 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8d74ba3caad418e85f936e3ad0e6c7e [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:06:47.169428 412 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8d74ba3caad418e85f936e3ad0e6c7e [term 1 LEADER]: Becoming Leader. State: Replica: c8d74ba3caad418e85f936e3ad0e6c7e, State: Running, Role: LEADER
I20240507 02:06:47.169428 407 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8d74ba3caad418e85f936e3ad0e6c7e [sys.catalog]: configured and running, proceeding with master startup.
I20240507 02:06:47.169559 412 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8d74ba3caad418e85f936e3ad0e6c7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8d74ba3caad418e85f936e3ad0e6c7e" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 40483 } }
I20240507 02:06:47.169849 414 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8d74ba3caad418e85f936e3ad0e6c7e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8d74ba3caad418e85f936e3ad0e6c7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8d74ba3caad418e85f936e3ad0e6c7e" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 40483 } } }
I20240507 02:06:47.169920 414 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8d74ba3caad418e85f936e3ad0e6c7e [sys.catalog]: This master's current role is: LEADER
I20240507 02:06:47.170037 414 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8d74ba3caad418e85f936e3ad0e6c7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8d74ba3caad418e85f936e3ad0e6c7e. Latest consensus state: current_term: 1 leader_uuid: "c8d74ba3caad418e85f936e3ad0e6c7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8d74ba3caad418e85f936e3ad0e6c7e" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 40483 } } }
I20240507 02:06:47.170084 414 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8d74ba3caad418e85f936e3ad0e6c7e [sys.catalog]: This master's current role is: LEADER
I20240507 02:06:47.170370 418 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 02:06:47.170830 418 catalog_manager.cc:1484] Initializing Kudu cluster ID...
May 07 02:06:47 dist-test-slave-v0bg krb5kdc[326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047607, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:06:47.174763 411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:47.165879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49232 (local address 127.31.131.126:40483)
0507 02:06:47.166006 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:47.166016 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:47.166036 (+ 20us) server_negotiation.cc:407] Connection header received
0507 02:06:47.166596 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:47.166605 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:47.166654 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:47.166762 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:06:47.167588 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:47.168279 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:47.168864 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:47.169093 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:47.172075 (+ 2982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:06:47.172081 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:06:47.172081 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:06:47.172093 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:06:47.173574 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:47.173914 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:47.173916 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:47.173916 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:47.173953 (+ 37us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:47.174086 (+ 133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:47.174086 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:47.174087 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:47.174240 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:06:47.174351 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:47.174459 (+ 108us) server_negotiation.cc:299] Negotiation successful
0507 02:06:47.174542 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":59,"threads_started":1}
I20240507 02:06:47.175247 418 catalog_manager.cc:1347] Generated new cluster ID: 0ac81dd708524c728f4773d3651be897
I20240507 02:06:47.175283 418 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240507 02:06:47.184358 418 catalog_manager.cc:1370] Generated new certificate authority record
I20240507 02:06:47.185031 418 catalog_manager.cc:1504] Loading token signing keys...
I20240507 02:06:47.209237 418 catalog_manager.cc:5857] T 00000000000000000000000000000000 P c8d74ba3caad418e85f936e3ad0e6c7e: Generated new TSK 0
I20240507 02:06:47.209439 418 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.131.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab.
Entry for principal kudu/127.31.131.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab.
I20240507 02:06:47.231336 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.65:0
--local_ip_for_outbound_sockets=127.31.131.65
--webserver_interface=127.31.131.65
--webserver_port=0
--tserver_master_addrs=127.31.131.126:40483
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab
--principal=kudu/127.31.131.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:37469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:06:47.305260 434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:06:47.305416 434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:06:47.305438 434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:06:47.306691 434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:06:47.306728 434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:06:47.306751 434 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.65
I20240507 02:06:47.308246 434 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:37469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab
--principal=kudu/127.31.131.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.131.65
--webserver_port=0
--tserver_master_addrs=127.31.131.126:40483
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.434
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.65
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:06:47.308413 434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:06:47.308569 434 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:06:47.312101 440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:06:47.312249 434 server_base.cc:1034] running on GCE node
W20240507 02:06:47.312115 442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:06:47.312332 439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:06:47.312531 434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:06:47.312729 434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:06:47.313851 434 hybrid_clock.cc:648] HybridClock initialized: now 1715047607313842 us; error 29 us; skew 500 ppm
May 07 02:06:47 dist-test-slave-v0bg krb5kdc[326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047607, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:06:47.316318 434 init.cc:377] Logged in from keytab as kudu/127.31.131.65@KRBTEST.COM (short username kudu)
I20240507 02:06:47.316773 434 webserver.cc:460] Webserver started at http://127.31.131.65:36573/ using document root <none> and password file <none>
I20240507 02:06:47.316956 434 fs_manager.cc:359] Metadata directory not provided
I20240507 02:06:47.317000 434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:06:47.317101 434 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:06:47.318250 434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/minicluster-data/ts-0/data/instance:
uuid: "2febe892bfcf4ae4bed4897cbdf30477"
format_stamp: "Formatted at 2024-05-07 02:06:47 on dist-test-slave-v0bg"
server_key: "3f5a143a66ed6cd01ba0c539d074e7d4"
server_key_iv: "2a1be4aad9d5f4b68e8b6efc54fbeaaa"
server_key_version: "encryptionkey@0"
I20240507 02:06:47.318594 434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/minicluster-data/ts-0/wal/instance:
uuid: "2febe892bfcf4ae4bed4897cbdf30477"
format_stamp: "Formatted at 2024-05-07 02:06:47 on dist-test-slave-v0bg"
server_key: "3f5a143a66ed6cd01ba0c539d074e7d4"
server_key_iv: "2a1be4aad9d5f4b68e8b6efc54fbeaaa"
server_key_version: "encryptionkey@0"
I20240507 02:06:47.319860 434 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240507 02:06:47.320554 450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:47.320725 434 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240507 02:06:47.320794 434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/minicluster-data/ts-0/wal
uuid: "2febe892bfcf4ae4bed4897cbdf30477"
format_stamp: "Formatted at 2024-05-07 02:06:47 on dist-test-slave-v0bg"
server_key: "3f5a143a66ed6cd01ba0c539d074e7d4"
server_key_iv: "2a1be4aad9d5f4b68e8b6efc54fbeaaa"
server_key_version: "encryptionkey@0"
I20240507 02:06:47.320858 434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:06:47.332298 434 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:06:47.333705 434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:06:47.333801 434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:06:47.333971 434 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:06:47.334232 434 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:06:47.334264 434 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:47.334282 434 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:06:47.334306 434 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:47.338979 434 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.65:43633
I20240507 02:06:47.339030 564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.65:43633 every 8 connection(s)
I20240507 02:06:47.339264 434 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
May 07 02:06:47 dist-test-slave-v0bg krb5kdc[326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047607, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.65@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:06:47.344687 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 434
I20240507 02:06:47.344765 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/minicluster-data/ts-0/wal/instance
I20240507 02:06:47.344928 32269 external_mini_cluster.cc:1428] Setting key 15703e104cc746fa318aef13fa5ecdfe
I20240507 02:06:47.346926 411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:47.339844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:36347 (local address 127.31.131.126:40483)
0507 02:06:47.339891 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:47.339893 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:47.340047 (+ 154us) server_negotiation.cc:407] Connection header received
0507 02:06:47.340498 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:47.340499 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:47.340511 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:47.340561 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:06:47.341250 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:47.341730 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:47.342241 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:47.342408 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:47.344245 (+ 1837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:06:47.344248 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:06:47.344248 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:06:47.344254 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:06:47.345549 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:47.345985 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:47.345986 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:47.345986 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:47.346004 (+ 18us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:47.346207 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:47.346208 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:47.346208 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:47.346300 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:06:47.346342 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:47.346717 (+ 375us) server_negotiation.cc:299] Negotiation successful
0507 02:06:47.346832 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20240507 02:06:47.347055 567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:47.339806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40483 (local address 127.31.131.65:36347)
0507 02:06:47.339949 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0507 02:06:47.339975 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0507 02:06:47.340404 (+ 429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:06:47.340626 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:06:47.340630 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:06:47.340893 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:06:47.341160 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:47.341164 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:47.341805 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:06:47.341806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:06:47.342169 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:47.342171 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:47.342507 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:47.342943 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:06:47.342956 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:06:47.344163 (+ 1207us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:06:47.345632 (+ 1469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:06:47.345638 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:06:47.345640 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:06:47.345921 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:06:47.346042 (+ 121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:06:47.346042 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:06:47.346043 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:06:47.346131 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:06:47.346351 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:06:47.346355 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:06:47.346530 (+ 175us) client_negotiation.cc:769] Sending connection context
0507 02:06:47.346709 (+ 179us) client_negotiation.cc:240] Negotiation successful
0507 02:06:47.346881 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":72,"threads_started":1}
I20240507 02:06:47.347486 565 heartbeater.cc:347] Connected to a master server at 127.31.131.126:40483
I20240507 02:06:47.347564 565 heartbeater.cc:464] Registering TS with master...
I20240507 02:06:47.347733 565 heartbeater.cc:511] Master 127.31.131.126:40483 requested a full tablet report, sending...
I20240507 02:06:47.348153 371 ts_manager.cc:194] Registered new tserver with Master: 2febe892bfcf4ae4bed4897cbdf30477 (127.31.131.65:43633)
I20240507 02:06:47.348747 371 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.131.65@KRBTEST.COM'} at 127.31.131.65:36347
WARNING: no policy specified for kudu/127.31.131.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab.
Entry for principal kudu/127.31.131.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab.
I20240507 02:06:47.365012 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.66:0
--local_ip_for_outbound_sockets=127.31.131.66
--webserver_interface=127.31.131.66
--webserver_port=0
--tserver_master_addrs=127.31.131.126:40483
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab
--principal=kudu/127.31.131.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:37469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:06:47.438879 570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:06:47.439025 570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:06:47.439046 570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:06:47.440294 570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:06:47.440335 570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:06:47.440361 570 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.66
I20240507 02:06:47.441867 570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:37469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab
--principal=kudu/127.31.131.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.131.66
--webserver_port=0
--tserver_master_addrs=127.31.131.126:40483
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.66
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:06:47.442051 570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:06:47.442214 570 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:06:47.445717 575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:06:47.445756 578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:06:47.445844 570 server_base.cc:1034] running on GCE node
W20240507 02:06:47.446008 576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:06:47.446208 570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:06:47.446388 570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:06:47.447515 570 hybrid_clock.cc:648] HybridClock initialized: now 1715047607447505 us; error 24 us; skew 500 ppm
May 07 02:06:47 dist-test-slave-v0bg krb5kdc[326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047607, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:06:47.449615 570 init.cc:377] Logged in from keytab as kudu/127.31.131.66@KRBTEST.COM (short username kudu)
I20240507 02:06:47.449981 570 webserver.cc:460] Webserver started at http://127.31.131.66:42975/ using document root <none> and password file <none>
I20240507 02:06:47.450115 570 fs_manager.cc:359] Metadata directory not provided
I20240507 02:06:47.450147 570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:06:47.450222 570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:06:47.451084 570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/minicluster-data/ts-1/data/instance:
uuid: "a3bdc9d509134c02b434056ad82939c8"
format_stamp: "Formatted at 2024-05-07 02:06:47 on dist-test-slave-v0bg"
server_key: "a04601ebe54a103802ec8bba29e79721"
server_key_iv: "d50b4386c6b7dde54742f166a79b8943"
server_key_version: "encryptionkey@0"
I20240507 02:06:47.451323 570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/minicluster-data/ts-1/wal/instance:
uuid: "a3bdc9d509134c02b434056ad82939c8"
format_stamp: "Formatted at 2024-05-07 02:06:47 on dist-test-slave-v0bg"
server_key: "a04601ebe54a103802ec8bba29e79721"
server_key_iv: "d50b4386c6b7dde54742f166a79b8943"
server_key_version: "encryptionkey@0"
I20240507 02:06:47.452302 570 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240507 02:06:47.452839 585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:47.452980 570 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240507 02:06:47.453044 570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/minicluster-data/ts-1/wal
uuid: "a3bdc9d509134c02b434056ad82939c8"
format_stamp: "Formatted at 2024-05-07 02:06:47 on dist-test-slave-v0bg"
server_key: "a04601ebe54a103802ec8bba29e79721"
server_key_iv: "d50b4386c6b7dde54742f166a79b8943"
server_key_version: "encryptionkey@0"
I20240507 02:06:47.453104 570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:06:47.478032 570 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:06:47.479437 570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:06:47.479527 570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:06:47.479696 570 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:06:47.479954 570 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:06:47.479985 570 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:47.480010 570 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:06:47.480026 570 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:47.484645 570 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.66:44681
I20240507 02:06:47.484678 700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.66:44681 every 8 connection(s)
I20240507 02:06:47.484918 570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
I20240507 02:06:47.488147 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 570
I20240507 02:06:47.488221 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/minicluster-data/ts-1/wal/instance
I20240507 02:06:47.488353 32269 external_mini_cluster.cc:1428] Setting key 8a6c2bc1cf603a1228c6a19003cdbd0b
May 07 02:06:47 dist-test-slave-v0bg krb5kdc[326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047607, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.66@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:06:47.492656 411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:47.485508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.66:38605 (local address 127.31.131.126:40483)
0507 02:06:47.485574 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:47.485576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:47.485705 (+ 129us) server_negotiation.cc:407] Connection header received
0507 02:06:47.486123 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:47.486124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:47.486136 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:47.486198 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:06:47.486901 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:47.487400 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:47.487915 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:47.488094 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:47.490268 (+ 2174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:06:47.490272 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:06:47.490272 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:06:47.490279 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:06:47.491515 (+ 1236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:47.491942 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:47.491943 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:47.491944 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:47.491963 (+ 19us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:47.492135 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:47.492136 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:47.492136 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:47.492219 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:06:47.492261 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:47.492534 (+ 273us) server_negotiation.cc:299] Negotiation successful
0507 02:06:47.492594 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20240507 02:06:47.492848 703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:47.485464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40483 (local address 127.31.131.66:38605)
0507 02:06:47.485603 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0507 02:06:47.485624 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0507 02:06:47.486030 (+ 406us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:06:47.486255 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:06:47.486258 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:06:47.486512 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:06:47.486813 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:47.486817 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:47.487473 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:06:47.487475 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:06:47.487837 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:47.487839 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:47.488195 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:47.488684 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:06:47.488702 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:06:47.490163 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:06:47.491613 (+ 1450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:06:47.491619 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:06:47.491621 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:06:47.491874 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:06:47.492015 (+ 141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:06:47.492015 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:06:47.492016 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:06:47.492087 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:06:47.492269 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:06:47.492273 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:06:47.492408 (+ 135us) client_negotiation.cc:769] Sending connection context
0507 02:06:47.492527 (+ 119us) client_negotiation.cc:240] Negotiation successful
0507 02:06:47.492648 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":73,"threads_started":1}
I20240507 02:06:47.493335 701 heartbeater.cc:347] Connected to a master server at 127.31.131.126:40483
I20240507 02:06:47.493415 701 heartbeater.cc:464] Registering TS with master...
I20240507 02:06:47.493572 701 heartbeater.cc:511] Master 127.31.131.126:40483 requested a full tablet report, sending...
I20240507 02:06:47.493918 371 ts_manager.cc:194] Registered new tserver with Master: a3bdc9d509134c02b434056ad82939c8 (127.31.131.66:44681)
I20240507 02:06:47.494285 371 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.131.66@KRBTEST.COM'} at 127.31.131.66:38605
WARNING: no policy specified for kudu/127.31.131.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab.
Entry for principal kudu/127.31.131.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab.
I20240507 02:06:47.509091 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.67:0
--local_ip_for_outbound_sockets=127.31.131.67
--webserver_interface=127.31.131.67
--webserver_port=0
--tserver_master_addrs=127.31.131.126:40483
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab
--principal=kudu/127.31.131.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:37469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:06:47.584528 706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:06:47.584692 706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:06:47.584721 706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:06:47.585944 706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:06:47.585984 706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:06:47.586015 706 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.67
I20240507 02:06:47.587512 706 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:37469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab
--principal=kudu/127.31.131.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.131.67
--webserver_port=0
--tserver_master_addrs=127.31.131.126:40483
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.67
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:06:47.587707 706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:06:47.587867 706 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240507 02:06:47.591495 706 server_base.cc:1034] running on GCE node
W20240507 02:06:47.591447 712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:06:47.591465 711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:06:47.591598 714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:06:47.591779 706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:06:47.591975 706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:06:47.593106 706 hybrid_clock.cc:648] HybridClock initialized: now 1715047607593091 us; error 28 us; skew 500 ppm
May 07 02:06:47 dist-test-slave-v0bg krb5kdc[326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047607, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:06:47.595198 706 init.cc:377] Logged in from keytab as kudu/127.31.131.67@KRBTEST.COM (short username kudu)
I20240507 02:06:47.595528 706 webserver.cc:460] Webserver started at http://127.31.131.67:42795/ using document root <none> and password file <none>
I20240507 02:06:47.595685 706 fs_manager.cc:359] Metadata directory not provided
I20240507 02:06:47.595726 706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:06:47.595820 706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:06:47.596709 706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/minicluster-data/ts-2/data/instance:
uuid: "444408a660bc47a5a9f69dfe10763b54"
format_stamp: "Formatted at 2024-05-07 02:06:47 on dist-test-slave-v0bg"
server_key: "379ec84255d65871be3ece513448cf01"
server_key_iv: "d21acca88df05f757ef80c18db85b623"
server_key_version: "encryptionkey@0"
I20240507 02:06:47.596982 706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/minicluster-data/ts-2/wal/instance:
uuid: "444408a660bc47a5a9f69dfe10763b54"
format_stamp: "Formatted at 2024-05-07 02:06:47 on dist-test-slave-v0bg"
server_key: "379ec84255d65871be3ece513448cf01"
server_key_iv: "d21acca88df05f757ef80c18db85b623"
server_key_version: "encryptionkey@0"
I20240507 02:06:47.597985 706 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240507 02:06:47.598501 721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:47.598645 706 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:47.598721 706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/minicluster-data/ts-2/wal
uuid: "444408a660bc47a5a9f69dfe10763b54"
format_stamp: "Formatted at 2024-05-07 02:06:47 on dist-test-slave-v0bg"
server_key: "379ec84255d65871be3ece513448cf01"
server_key_iv: "d21acca88df05f757ef80c18db85b623"
server_key_version: "encryptionkey@0"
I20240507 02:06:47.598773 706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:06:47.615978 706 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:06:47.617404 706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:06:47.617507 706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:06:47.617681 706 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:06:47.617944 706 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:06:47.617972 706 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:47.618000 706 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:06:47.618027 706 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:47.622679 706 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.67:45375
I20240507 02:06:47.622725 835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.67:45375 every 8 connection(s)
I20240507 02:06:47.622972 706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
May 07 02:06:47 dist-test-slave-v0bg krb5kdc[326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047607, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.67@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:06:47.629859 411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:47.623554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.67:44871 (local address 127.31.131.126:40483)
0507 02:06:47.623607 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:47.623610 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:47.623768 (+ 158us) server_negotiation.cc:407] Connection header received
0507 02:06:47.624240 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:47.624242 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:47.624252 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:47.624297 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:06:47.624975 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:47.625451 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:47.625956 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:47.626120 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:47.627984 (+ 1864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:06:47.627988 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:06:47.627988 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:06:47.627994 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:06:47.629015 (+ 1021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:47.629311 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:47.629311 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:47.629312 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:47.629325 (+ 13us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:47.629450 (+ 125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:47.629450 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:47.629451 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:47.629523 (+ 72us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:06:47.629563 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:47.629745 (+ 182us) server_negotiation.cc:299] Negotiation successful
0507 02:06:47.629797 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20240507 02:06:47.629952 838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:47.623528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40483 (local address 127.31.131.67:44871)
0507 02:06:47.623676 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0507 02:06:47.623699 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0507 02:06:47.624136 (+ 437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:06:47.624361 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:06:47.624365 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:06:47.624626 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:06:47.624884 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:47.624887 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:47.625522 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:06:47.625524 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:06:47.625883 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:47.625885 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:47.626230 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:47.626704 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:06:47.626717 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:06:47.627909 (+ 1192us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:06:47.629077 (+ 1168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:06:47.629081 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:06:47.629084 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:06:47.629259 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:06:47.629369 (+ 110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:06:47.629370 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:06:47.629370 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:06:47.629413 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:06:47.629561 (+ 148us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:06:47.629563 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:06:47.629654 (+ 91us) client_negotiation.cc:769] Sending connection context
0507 02:06:47.629730 (+ 76us) client_negotiation.cc:240] Negotiation successful
0507 02:06:47.629819 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":74,"threads_started":1}
I20240507 02:06:47.630347 836 heartbeater.cc:347] Connected to a master server at 127.31.131.126:40483
I20240507 02:06:47.630429 836 heartbeater.cc:464] Registering TS with master...
I20240507 02:06:47.630628 836 heartbeater.cc:511] Master 127.31.131.126:40483 requested a full tablet report, sending...
I20240507 02:06:47.630980 371 ts_manager.cc:194] Registered new tserver with Master: 444408a660bc47a5a9f69dfe10763b54 (127.31.131.67:45375)
I20240507 02:06:47.631332 371 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.131.67@KRBTEST.COM'} at 127.31.131.67:44871
I20240507 02:06:47.632287 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 706
I20240507 02:06:47.632362 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1715047604443657-32269-0/minicluster-data/ts-2/wal/instance
I20240507 02:06:47.632513 32269 external_mini_cluster.cc:1428] Setting key 1db4e2687ffc725b9414e47b1e62e52b
I20240507 02:06:47.633230 32269 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240507 02:06:47.639019 411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:47.634498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49236 (local address 127.31.131.126:40483)
0507 02:06:47.634543 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:47.634544 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:47.634685 (+ 141us) server_negotiation.cc:407] Connection header received
0507 02:06:47.634738 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:47.634739 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:47.634748 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:47.634793 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:06:47.635398 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:47.635838 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:47.636341 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:47.636490 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:47.637227 (+ 737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:06:47.637230 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:06:47.637230 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:06:47.637235 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:06:47.638166 (+ 931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:47.638411 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:47.638412 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:47.638413 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:47.638432 (+ 19us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:47.638592 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:47.638593 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:47.638593 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:47.638694 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:06:47.638780 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:47.638869 (+ 89us) server_negotiation.cc:299] Negotiation successful
0507 02:06:47.638935 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20240507 02:06:47.640741 371 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49236:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 02:06:47.642081 371 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 02:06:47.646842 845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:47.643761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.66:44681 (local address 127.0.0.1:43254)
0507 02:06:47.643909 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0507 02:06:47.643928 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0507 02:06:47.643989 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:06:47.644381 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:06:47.644384 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:06:47.644386 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:06:47.644585 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:47.644586 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:47.646006 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:06:47.646007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:06:47.646593 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:47.646595 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:47.646662 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:47.646667 (+ 5us) client_negotiation.cc:769] Sending connection context
0507 02:06:47.646708 (+ 41us) client_negotiation.cc:240] Negotiation successful
0507 02:06:47.646744 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":57,"threads_started":1}
I20240507 02:06:47.647015 844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:47.643764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.65:43633 (local address 127.0.0.1:49992)
0507 02:06:47.644008 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0507 02:06:47.644018 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0507 02:06:47.644062 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:06:47.644526 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:06:47.644527 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:06:47.644529 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:06:47.644690 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:47.644691 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:47.645972 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:06:47.645973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:06:47.646813 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:47.646817 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:47.646872 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:47.646874 (+ 2us) client_negotiation.cc:769] Sending connection context
0507 02:06:47.646900 (+ 26us) client_negotiation.cc:240] Negotiation successful
0507 02:06:47.646934 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":89,"threads_started":1}
I20240507 02:06:47.646859 847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:47.643958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.67:45375 (local address 127.0.0.1:60300)
0507 02:06:47.644133 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0507 02:06:47.644141 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0507 02:06:47.644176 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:06:47.644753 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:06:47.644754 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:06:47.644755 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:06:47.644904 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:47.644906 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:47.645703 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:06:47.645705 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:06:47.646524 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:47.646526 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:47.646621 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:47.646653 (+ 32us) client_negotiation.cc:769] Sending connection context
0507 02:06:47.646696 (+ 43us) client_negotiation.cc:240] Negotiation successful
0507 02:06:47.646733 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":57,"threads_started":1}
I20240507 02:06:47.647604 846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:47.643861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49992 (local address 127.31.131.65:43633)
0507 02:06:47.644386 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:47.644392 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:47.644400 (+ 8us) server_negotiation.cc:407] Connection header received
0507 02:06:47.644434 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:47.644435 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:47.644449 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:47.644522 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:06:47.644749 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:47.645899 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:47.646989 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:47.647468 (+ 479us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:47.647501 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:47.647526 (+ 25us) server_negotiation.cc:299] Negotiation successful
0507 02:06:47.647550 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":79,"threads_started":1}
I20240507 02:06:47.647769 849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:47.644237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60300 (local address 127.31.131.67:45375)
0507 02:06:47.644374 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:47.644382 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:47.644392 (+ 10us) server_negotiation.cc:407] Connection header received
0507 02:06:47.644424 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:47.644426 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:47.644440 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:47.644522 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:06:47.644978 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:47.645630 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:47.646865 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:47.647606 (+ 741us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:47.647638 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:47.647663 (+ 25us) server_negotiation.cc:299] Negotiation successful
0507 02:06:47.647700 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":64,"threads_started":1}
I20240507 02:06:47.647948 848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:47.644076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43254 (local address 127.31.131.66:44681)
0507 02:06:47.644244 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:47.644250 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:47.644257 (+ 7us) server_negotiation.cc:407] Connection header received
0507 02:06:47.644288 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:47.644289 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:47.644302 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:47.644368 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:06:47.644670 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:47.645928 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:47.647014 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:47.647581 (+ 567us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:47.647625 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:47.647827 (+ 202us) server_negotiation.cc:299] Negotiation successful
0507 02:06:47.647877 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":99,"threads_started":1}
I20240507 02:06:47.648161 498 tablet_service.cc:1467] Processing CreateTablet for tablet 29ce4936680f4bafbde20a617b48fad4 (DEFAULT_TABLE table=test-table [id=9186d9699e5648d2a51403a705fd2890]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:06:47.648337 633 tablet_service.cc:1467] Processing CreateTablet for tablet 29ce4936680f4bafbde20a617b48fad4 (DEFAULT_TABLE table=test-table [id=9186d9699e5648d2a51403a705fd2890]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:06:47.648337 769 tablet_service.cc:1467] Processing CreateTablet for tablet 29ce4936680f4bafbde20a617b48fad4 (DEFAULT_TABLE table=test-table [id=9186d9699e5648d2a51403a705fd2890]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:06:47.648425 498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29ce4936680f4bafbde20a617b48fad4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:06:47.648588 769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29ce4936680f4bafbde20a617b48fad4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:06:47.648588 633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29ce4936680f4bafbde20a617b48fad4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:06:47.650664 850 tablet_bootstrap.cc:492] T 29ce4936680f4bafbde20a617b48fad4 P 2febe892bfcf4ae4bed4897cbdf30477: Bootstrap starting.
I20240507 02:06:47.651105 850 tablet_bootstrap.cc:654] T 29ce4936680f4bafbde20a617b48fad4 P 2febe892bfcf4ae4bed4897cbdf30477: Neither blocks nor log segments found. Creating new log.
I20240507 02:06:47.651373 850 log.cc:824] T 29ce4936680f4bafbde20a617b48fad4 P 2febe892bfcf4ae4bed4897cbdf30477: Log is configured to *not* fsync() on all Append() calls
I20240507 02:06:47.651593 852 tablet_bootstrap.cc:492] T 29ce4936680f4bafbde20a617b48fad4 P a3bdc9d509134c02b434056ad82939c8: Bootstrap starting.
I20240507 02:06:47.651993 850 tablet_bootstrap.cc:492] T 29ce4936680f4bafbde20a617b48fad4 P 2febe892bfcf4ae4bed4897cbdf30477: No bootstrap required, opened a new log
I20240507 02:06:47.652036 852 tablet_bootstrap.cc:654] T 29ce4936680f4bafbde20a617b48fad4 P a3bdc9d509134c02b434056ad82939c8: Neither blocks nor log segments found. Creating new log.
I20240507 02:06:47.652045 850 ts_tablet_manager.cc:1367] T 29ce4936680f4bafbde20a617b48fad4 P 2febe892bfcf4ae4bed4897cbdf30477: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240507 02:06:47.652259 852 log.cc:824] T 29ce4936680f4bafbde20a617b48fad4 P a3bdc9d509134c02b434056ad82939c8: Log is configured to *not* fsync() on all Append() calls
I20240507 02:06:47.652529 851 tablet_bootstrap.cc:492] T 29ce4936680f4bafbde20a617b48fad4 P 444408a660bc47a5a9f69dfe10763b54: Bootstrap starting.
I20240507 02:06:47.652941 851 tablet_bootstrap.cc:654] T 29ce4936680f4bafbde20a617b48fad4 P 444408a660bc47a5a9f69dfe10763b54: Neither blocks nor log segments found. Creating new log.
I20240507 02:06:47.652966 852 tablet_bootstrap.cc:492] T 29ce4936680f4bafbde20a617b48fad4 P a3bdc9d509134c02b434056ad82939c8: No bootstrap required, opened a new log
I20240507 02:06:47.653015 852 ts_tablet_manager.cc:1367] T 29ce4936680f4bafbde20a617b48fad4 P a3bdc9d509134c02b434056ad82939c8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240507 02:06:47.653172 851 log.cc:824] T 29ce4936680f4bafbde20a617b48fad4 P 444408a660bc47a5a9f69dfe10763b54: Log is configured to *not* fsync() on all Append() calls
I20240507 02:06:47.653728 851 tablet_bootstrap.cc:492] T 29ce4936680f4bafbde20a617b48fad4 P 444408a660bc47a5a9f69dfe10763b54: No bootstrap required, opened a new log
I20240507 02:06:47.653779 851 ts_tablet_manager.cc:1367] T 29ce4936680f4bafbde20a617b48fad4 P 444408a660bc47a5a9f69dfe10763b54: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240507 02:06:47.654096 850 raft_consensus.cc:348] T 29ce4936680f4bafbde20a617b48fad4 P 2febe892bfcf4ae4bed4897cbdf30477 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3bdc9d509134c02b434056ad82939c8" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44681 } } peers { permanent_uuid: "2febe892bfcf4ae4bed4897cbdf30477" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43633 } } peers { permanent_uuid: "444408a660bc47a5a9f69dfe10763b54" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 45375 } }
I20240507 02:06:47.654225 850 raft_consensus.cc:374] T 29ce4936680f4bafbde20a617b48fad4 P 2febe892bfcf4ae4bed4897cbdf30477 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:06:47.654250 850 raft_consensus.cc:729] T 29ce4936680f4bafbde20a617b48fad4 P 2febe892bfcf4ae4bed4897cbdf30477 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2febe892bfcf4ae4bed4897cbdf30477, State: Initialized, Role: FOLLOWER
I20240507 02:06:47.654376 850 consensus_queue.cc:260] T 29ce4936680f4bafbde20a617b48fad4 P 2febe892bfcf4ae4bed4897cbdf30477 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3bdc9d509134c02b434056ad82939c8" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44681 } } peers { permanent_uuid: "2febe892bfcf4ae4bed4897cbdf30477" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43633 } } peers { permanent_uuid: "444408a660bc47a5a9f69dfe10763b54" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 45375 } }
I20240507 02:06:47.654639 850 ts_tablet_manager.cc:1398] T 29ce4936680f4bafbde20a617b48fad4 P 2febe892bfcf4ae4bed4897cbdf30477: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240507 02:06:47.654762 565 heartbeater.cc:503] Master 127.31.131.126:40483 was elected leader, sending a full tablet report...
I20240507 02:06:47.655154 852 raft_consensus.cc:348] T 29ce4936680f4bafbde20a617b48fad4 P a3bdc9d509134c02b434056ad82939c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3bdc9d509134c02b434056ad82939c8" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44681 } } peers { permanent_uuid: "2febe892bfcf4ae4bed4897cbdf30477" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43633 } } peers { permanent_uuid: "444408a660bc47a5a9f69dfe10763b54" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 45375 } }
I20240507 02:06:47.655256 852 raft_consensus.cc:374] T 29ce4936680f4bafbde20a617b48fad4 P a3bdc9d509134c02b434056ad82939c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:06:47.655275 852 raft_consensus.cc:729] T 29ce4936680f4bafbde20a617b48fad4 P a3bdc9d509134c02b434056ad82939c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3bdc9d509134c02b434056ad82939c8, State: Initialized, Role: FOLLOWER
I20240507 02:06:47.655387 852 consensus_queue.cc:260] T 29ce4936680f4bafbde20a617b48fad4 P a3bdc9d509134c02b434056ad82939c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3bdc9d509134c02b434056ad82939c8" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44681 } } peers { permanent_uuid: "2febe892bfcf4ae4bed4897cbdf30477" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43633 } } peers { permanent_uuid: "444408a660bc47a5a9f69dfe10763b54" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 45375 } }
I20240507 02:06:47.655601 852 ts_tablet_manager.cc:1398] T 29ce4936680f4bafbde20a617b48fad4 P a3bdc9d509134c02b434056ad82939c8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240507 02:06:47.655697 701 heartbeater.cc:503] Master 127.31.131.126:40483 was elected leader, sending a full tablet report...
I20240507 02:06:47.656265 851 raft_consensus.cc:348] T 29ce4936680f4bafbde20a617b48fad4 P 444408a660bc47a5a9f69dfe10763b54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3bdc9d509134c02b434056ad82939c8" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44681 } } peers { permanent_uuid: "2febe892bfcf4ae4bed4897cbdf30477" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43633 } } peers { permanent_uuid: "444408a660bc47a5a9f69dfe10763b54" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 45375 } }
I20240507 02:06:47.656364 851 raft_consensus.cc:374] T 29ce4936680f4bafbde20a617b48fad4 P 444408a660bc47a5a9f69dfe10763b54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:06:47.656378 851 raft_consensus.cc:729] T 29ce4936680f4bafbde20a617b48fad4 P 444408a660bc47a5a9f69dfe10763b54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 444408a660bc47a5a9f69dfe10763b54, State: Initialized, Role: FOLLOWER
I20240507 02:06:47.656431 851 consensus_queue.cc:260] T 29ce4936680f4bafbde20a617b48fad4 P 444408a660bc47a5a9f69dfe10763b54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3bdc9d509134c02b434056ad82939c8" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44681 } } peers { permanent_uuid: "2febe892bfcf4ae4bed4897cbdf30477" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43633 } } peers { permanent_uuid: "444408a660bc47a5a9f69dfe10763b54" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 45375 } }
I20240507 02:06:47.656607 851 ts_tablet_manager.cc:1398] T 29ce4936680f4bafbde20a617b48fad4 P 444408a660bc47a5a9f69dfe10763b54: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240507 02:06:47.656653 836 heartbeater.cc:503] Master 127.31.131.126:40483 was elected leader, sending a full tablet report...
W20240507 02:06:47.735358 702 tablet.cc:2369] T 29ce4936680f4bafbde20a617b48fad4 P a3bdc9d509134c02b434056ad82939c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240507 02:06:47.761902 858 raft_consensus.cc:482] T 29ce4936680f4bafbde20a617b48fad4 P 444408a660bc47a5a9f69dfe10763b54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 02:06:47.762010 858 raft_consensus.cc:504] T 29ce4936680f4bafbde20a617b48fad4 P 444408a660bc47a5a9f69dfe10763b54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3bdc9d509134c02b434056ad82939c8" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44681 } } peers { permanent_uuid: "2febe892bfcf4ae4bed4897cbdf30477" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43633 } } peers { permanent_uuid: "444408a660bc47a5a9f69dfe10763b54" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 45375 } }
I20240507 02:06:47.762328 858 leader_election.cc:288] T 29ce4936680f4bafbde20a617b48fad4 P 444408a660bc47a5a9f69dfe10763b54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3bdc9d509134c02b434056ad82939c8 (127.31.131.66:44681), 2febe892bfcf4ae4bed4897cbdf30477 (127.31.131.65:43633)
I20240507 02:06:47.765027 838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:47.762484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.65:43633 (local address 127.31.131.67:54463)
0507 02:06:47.762537 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0507 02:06:47.762563 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0507 02:06:47.762601 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:06:47.762737 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:06:47.762738 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:06:47.762740 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:06:47.762965 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:47.762967 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:47.763866 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:06:47.763867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:06:47.764801 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:47.764803 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:47.764874 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:47.764877 (+ 3us) client_negotiation.cc:769] Sending connection context
0507 02:06:47.764907 (+ 30us) client_negotiation.cc:240] Negotiation successful
0507 02:06:47.764938 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20240507 02:06:47.765034 859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:47.762484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.66:44681 (local address 127.31.131.67:48883)
0507 02:06:47.762677 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0507 02:06:47.762683 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0507 02:06:47.762733 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:06:47.762837 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:06:47.762837 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:06:47.762838 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:06:47.763045 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:47.763046 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:47.763866 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:06:47.763867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:06:47.764805 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:47.764807 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:47.764873 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:47.764877 (+ 4us) client_negotiation.cc:769] Sending connection context
0507 02:06:47.764907 (+ 30us) client_negotiation.cc:240] Negotiation successful
0507 02:06:47.764938 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":17,"thread_start_us":114,"threads_started":1}
I20240507 02:06:47.765437 846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:47.762572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.67:54463 (local address 127.31.131.65:43633)
0507 02:06:47.762610 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:47.762612 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:47.762620 (+ 8us) server_negotiation.cc:407] Connection header received
0507 02:06:47.762650 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:47.762650 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:47.762658 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:47.762728 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:06:47.763038 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:47.763799 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:47.764877 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:47.765321 (+ 444us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:47.765329 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:47.765349 (+ 20us) server_negotiation.cc:299] Negotiation successful
0507 02:06:47.765374 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20240507 02:06:47.765437 848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:47.762538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.67:48883 (local address 127.31.131.66:44681)
0507 02:06:47.762600 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:47.762602 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:47.762741 (+ 139us) server_negotiation.cc:407] Connection header received
0507 02:06:47.762773 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:47.762774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:47.762783 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:47.762826 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:06:47.763102 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:47.763798 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:47.764877 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:47.765321 (+ 444us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:47.765329 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:47.765349 (+ 20us) server_negotiation.cc:299] Negotiation successful
0507 02:06:47.765375 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20240507 02:06:47.765738 653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29ce4936680f4bafbde20a617b48fad4" candidate_uuid: "444408a660bc47a5a9f69dfe10763b54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3bdc9d509134c02b434056ad82939c8" is_pre_election: true
I20240507 02:06:47.765736 518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29ce4936680f4bafbde20a617b48fad4" candidate_uuid: "444408a660bc47a5a9f69dfe10763b54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2febe892bfcf4ae4bed4897cbdf30477" is_pre_election: true
I20240507 02:06:47.765856 518 raft_consensus.cc:2454] T 29ce4936680f4bafbde20a617b48fad4 P 2febe892bfcf4ae4bed4897cbdf30477 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 444408a660bc47a5a9f69dfe10763b54 in term 0.
I20240507 02:06:47.765863 653 raft_consensus.cc:2454] T 29ce4936680f4bafbde20a617b48fad4 P a3bdc9d509134c02b434056ad82939c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 444408a660bc47a5a9f69dfe10763b54 in term 0.
I20240507 02:06:47.766067 722 leader_election.cc:302] T 29ce4936680f4bafbde20a617b48fad4 P 444408a660bc47a5a9f69dfe10763b54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 444408a660bc47a5a9f69dfe10763b54, a3bdc9d509134c02b434056ad82939c8; no voters:
I20240507 02:06:47.766186 858 raft_consensus.cc:2781] T 29ce4936680f4bafbde20a617b48fad4 P 444408a660bc47a5a9f69dfe10763b54 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 02:06:47.766239 858 raft_consensus.cc:482] T 29ce4936680f4bafbde20a617b48fad4 P 444408a660bc47a5a9f69dfe10763b54 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 02:06:47.766254 858 raft_consensus.cc:3037] T 29ce4936680f4bafbde20a617b48fad4 P 444408a660bc47a5a9f69dfe10763b54 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:06:47.766908 858 raft_consensus.cc:504] T 29ce4936680f4bafbde20a617b48fad4 P 444408a660bc47a5a9f69dfe10763b54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3bdc9d509134c02b434056ad82939c8" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44681 } } peers { permanent_uuid: "2febe892bfcf4ae4bed4897cbdf30477" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43633 } } peers { permanent_uuid: "444408a660bc47a5a9f69dfe10763b54" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 45375 } }
I20240507 02:06:47.767019 858 leader_election.cc:288] T 29ce4936680f4bafbde20a617b48fad4 P 444408a660bc47a5a9f69dfe10763b54 [CANDIDATE]: Term 1 election: Requested vote from peers a3bdc9d509134c02b434056ad82939c8 (127.31.131.66:44681), 2febe892bfcf4ae4bed4897cbdf30477 (127.31.131.65:43633)
I20240507 02:06:47.767158 518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29ce4936680f4bafbde20a617b48fad4" candidate_uuid: "444408a660bc47a5a9f69dfe10763b54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2febe892bfcf4ae4bed4897cbdf30477"
I20240507 02:06:47.767205 653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29ce4936680f4bafbde20a617b48fad4" candidate_uuid: "444408a660bc47a5a9f69dfe10763b54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3bdc9d509134c02b434056ad82939c8"
I20240507 02:06:47.767238 518 raft_consensus.cc:3037] T 29ce4936680f4bafbde20a617b48fad4 P 2febe892bfcf4ae4bed4897cbdf30477 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:06:47.767259 653 raft_consensus.cc:3037] T 29ce4936680f4bafbde20a617b48fad4 P a3bdc9d509134c02b434056ad82939c8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:06:47.768036 518 raft_consensus.cc:2454] T 29ce4936680f4bafbde20a617b48fad4 P 2febe892bfcf4ae4bed4897cbdf30477 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 444408a660bc47a5a9f69dfe10763b54 in term 1.
I20240507 02:06:47.768036 653 raft_consensus.cc:2454] T 29ce4936680f4bafbde20a617b48fad4 P a3bdc9d509134c02b434056ad82939c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 444408a660bc47a5a9f69dfe10763b54 in term 1.
I20240507 02:06:47.768193 722 leader_election.cc:302] T 29ce4936680f4bafbde20a617b48fad4 P 444408a660bc47a5a9f69dfe10763b54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 444408a660bc47a5a9f69dfe10763b54, a3bdc9d509134c02b434056ad82939c8; no voters:
I20240507 02:06:47.768311 858 raft_consensus.cc:2781] T 29ce4936680f4bafbde20a617b48fad4 P 444408a660bc47a5a9f69dfe10763b54 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:06:47.768442 858 raft_consensus.cc:686] T 29ce4936680f4bafbde20a617b48fad4 P 444408a660bc47a5a9f69dfe10763b54 [term 1 LEADER]: Becoming Leader. State: Replica: 444408a660bc47a5a9f69dfe10763b54, State: Running, Role: LEADER
I20240507 02:06:47.768532 858 consensus_queue.cc:237] T 29ce4936680f4bafbde20a617b48fad4 P 444408a660bc47a5a9f69dfe10763b54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3bdc9d509134c02b434056ad82939c8" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44681 } } peers { permanent_uuid: "2febe892bfcf4ae4bed4897cbdf30477" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43633 } } peers { permanent_uuid: "444408a660bc47a5a9f69dfe10763b54" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 45375 } }
I20240507 02:06:47.769238 370 catalog_manager.cc:5484] T 29ce4936680f4bafbde20a617b48fad4 P 444408a660bc47a5a9f69dfe10763b54 reported cstate change: term changed from 0 to 1, leader changed from <none> to 444408a660bc47a5a9f69dfe10763b54 (127.31.131.67). New cstate: current_term: 1 leader_uuid: "444408a660bc47a5a9f69dfe10763b54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3bdc9d509134c02b434056ad82939c8" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2febe892bfcf4ae4bed4897cbdf30477" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43633 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "444408a660bc47a5a9f69dfe10763b54" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 45375 } health_report { overall_health: HEALTHY } } }
May 07 02:06:47 dist-test-slave-v0bg krb5kdc[326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047607, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.131.65@KRBTEST.COM
I20240507 02:06:47.795603 846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:47.789146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49998 (local address 127.31.131.65:43633)
0507 02:06:47.789190 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:47.789192 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:47.789304 (+ 112us) server_negotiation.cc:407] Connection header received
0507 02:06:47.789364 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:47.789365 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:47.789374 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:47.789413 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:06:47.790023 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:47.790449 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:47.791022 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:47.791172 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:47.793465 (+ 2293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:06:47.793476 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:06:47.793477 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:06:47.793482 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:06:47.794763 (+ 1281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:47.795024 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:47.795027 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:47.795027 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:47.795059 (+ 32us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:47.795193 (+ 134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:47.795194 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:47.795195 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:47.795303 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:06:47.795427 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:47.795454 (+ 27us) server_negotiation.cc:299] Negotiation successful
0507 02:06:47.795526 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
May 07 02:06:47 dist-test-slave-v0bg krb5kdc[326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047607, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
May 07 02:06:47 dist-test-slave-v0bg krb5kdc[326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047607, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.131.65@KRBTEST.COM
I20240507 02:06:47.814448 846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:47.809004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50006 (local address 127.31.131.65:43633)
0507 02:06:47.809052 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:47.809053 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:47.809106 (+ 53us) server_negotiation.cc:407] Connection header received
0507 02:06:47.809148 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:47.809149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:47.809162 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:47.809209 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:06:47.809769 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:47.810210 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:47.810731 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:47.811204 (+ 473us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:47.812759 (+ 1555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:06:47.812763 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:06:47.812763 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:06:47.812768 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:06:47.813800 (+ 1032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:47.814026 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:47.814026 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:47.814027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:47.814040 (+ 13us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:47.814125 (+ 85us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:47.814125 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:47.814125 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:47.814188 (+ 63us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:06:47.814251 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:47.814335 (+ 84us) server_negotiation.cc:299] Negotiation successful
0507 02:06:47.814386 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
May 07 02:06:47 dist-test-slave-v0bg krb5kdc[326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047607, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
May 07 02:06:47 dist-test-slave-v0bg krb5kdc[326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047607, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.131.65@KRBTEST.COM
I20240507 02:06:47.832528 846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:47.827245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50008 (local address 127.31.131.65:43633)
0507 02:06:47.827297 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:47.827300 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:47.827360 (+ 60us) server_negotiation.cc:407] Connection header received
0507 02:06:47.827425 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:47.827426 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:47.827435 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:47.827479 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:06:47.828067 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:47.828504 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:47.829007 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:47.829168 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:47.830795 (+ 1627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:06:47.830798 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:06:47.830799 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:06:47.830804 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:06:47.831757 (+ 953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:47.831999 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:47.832000 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:47.832000 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:47.832021 (+ 21us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:47.832133 (+ 112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:47.832134 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:47.832134 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:47.832204 (+ 70us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:06:47.832285 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:47.832370 (+ 85us) server_negotiation.cc:299] Negotiation successful
0507 02:06:47.832438 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
W20240507 02:06:47.832932 478 tablet_service.cc:2908] Rejecting scan request for tablet 29ce4936680f4bafbde20a617b48fad4: Uninitialized: clean time has not yet been initialized
I20240507 02:06:47.833809 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 434
I20240507 02:06:47.838407 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 570
I20240507 02:06:47.842190 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 706
I20240507 02:06:47.845774 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 339
2024-05-07T02:06:47Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (1764 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 07 02:06:47 dist-test-slave-v0bg krb5kdc[874](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 07 02:06:47 dist-test-slave-v0bg krb5kdc[874](info): set up 2 sockets
May 07 02:06:47 dist-test-slave-v0bg krb5kdc[874](info): commencing operation
krb5kdc: starting...
W20240507 02:06:48.743266 32269 mini_kdc.cc:121] Time spent starting KDC: real 0.879s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/krb5kdc/test-user.keytab.
May 07 02:06:48 dist-test-slave-v0bg krb5kdc[874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047608, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-07T02:06:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-07T02:06:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.131.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab.
Entry for principal kudu/127.31.131.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab.
I20240507 02:06:48.842478 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:44459
--webserver_interface=127.31.131.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab
--principal=kudu/127.31.131.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:45031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.131.126:44459
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:06:48.916421 887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:06:48.916596 887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:06:48.916625 887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:06:48.917878 887 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240507 02:06:48.917917 887 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20240507 02:06:48.917933 887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:06:48.917949 887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:06:48.917963 887 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240507 02:06:48.917979 887 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240507 02:06:48.919509 887 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:45031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.131.126:44459
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab
--principal=kudu/127.31.131.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:44459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.131.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:06:48.919708 887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:06:48.919872 887 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:06:48.923389 893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:06:48.923501 887 server_base.cc:1034] running on GCE node
W20240507 02:06:48.923550 892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:06:48.923584 895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:06:48.923784 887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:06:48.923976 887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:06:48.925108 887 hybrid_clock.cc:648] HybridClock initialized: now 1715047608925096 us; error 27 us; skew 500 ppm
May 07 02:06:48 dist-test-slave-v0bg krb5kdc[874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047608, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:06:48.927225 887 init.cc:377] Logged in from keytab as kudu/127.31.131.126@KRBTEST.COM (short username kudu)
I20240507 02:06:48.927580 887 webserver.cc:460] Webserver started at http://127.31.131.126:45179/ using document root <none> and password file <none>
I20240507 02:06:48.927726 887 fs_manager.cc:359] Metadata directory not provided
I20240507 02:06:48.927762 887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:06:48.927858 887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:06:48.928737 887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/minicluster-data/master-0/data/instance:
uuid: "c10b07de71ff43efb4d7a1d211569a29"
format_stamp: "Formatted at 2024-05-07 02:06:48 on dist-test-slave-v0bg"
server_key: "5c5f2fc0177121406d56bd4027dca400"
server_key_iv: "f4726d90f5d612bc7b2f9c47079ead57"
server_key_version: "encryptionkey@0"
I20240507 02:06:48.929008 887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/minicluster-data/master-0/wal/instance:
uuid: "c10b07de71ff43efb4d7a1d211569a29"
format_stamp: "Formatted at 2024-05-07 02:06:48 on dist-test-slave-v0bg"
server_key: "5c5f2fc0177121406d56bd4027dca400"
server_key_iv: "f4726d90f5d612bc7b2f9c47079ead57"
server_key_version: "encryptionkey@0"
I20240507 02:06:48.930012 887 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240507 02:06:48.930588 902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:48.930733 887 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240507 02:06:48.930804 887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/minicluster-data/master-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/minicluster-data/master-0/wal
uuid: "c10b07de71ff43efb4d7a1d211569a29"
format_stamp: "Formatted at 2024-05-07 02:06:48 on dist-test-slave-v0bg"
server_key: "5c5f2fc0177121406d56bd4027dca400"
server_key_iv: "f4726d90f5d612bc7b2f9c47079ead57"
server_key_version: "encryptionkey@0"
I20240507 02:06:48.930861 887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:06:48.943409 887 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:06:48.945191 887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:06:48.945289 887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:06:48.949033 887 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.126:44459
I20240507 02:06:48.949095 965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.126:44459 every 8 connection(s)
I20240507 02:06:48.949352 887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
I20240507 02:06:48.949893 966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:06:48.951949 966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c10b07de71ff43efb4d7a1d211569a29: Bootstrap starting.
I20240507 02:06:48.952448 966 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c10b07de71ff43efb4d7a1d211569a29: Neither blocks nor log segments found. Creating new log.
I20240507 02:06:48.952682 966 log.cc:824] T 00000000000000000000000000000000 P c10b07de71ff43efb4d7a1d211569a29: Log is configured to *not* fsync() on all Append() calls
I20240507 02:06:48.953250 966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c10b07de71ff43efb4d7a1d211569a29: No bootstrap required, opened a new log
I20240507 02:06:48.954639 966 raft_consensus.cc:348] T 00000000000000000000000000000000 P c10b07de71ff43efb4d7a1d211569a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c10b07de71ff43efb4d7a1d211569a29" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 44459 } }
I20240507 02:06:48.954752 966 raft_consensus.cc:374] T 00000000000000000000000000000000 P c10b07de71ff43efb4d7a1d211569a29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:06:48.954767 966 raft_consensus.cc:729] T 00000000000000000000000000000000 P c10b07de71ff43efb4d7a1d211569a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c10b07de71ff43efb4d7a1d211569a29, State: Initialized, Role: FOLLOWER
I20240507 02:06:48.954881 966 consensus_queue.cc:260] T 00000000000000000000000000000000 P c10b07de71ff43efb4d7a1d211569a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c10b07de71ff43efb4d7a1d211569a29" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 44459 } }
I20240507 02:06:48.954931 966 raft_consensus.cc:388] T 00000000000000000000000000000000 P c10b07de71ff43efb4d7a1d211569a29 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 02:06:48.954962 966 raft_consensus.cc:482] T 00000000000000000000000000000000 P c10b07de71ff43efb4d7a1d211569a29 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 02:06:48.955000 966 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c10b07de71ff43efb4d7a1d211569a29 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:06:48.955535 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 887
I20240507 02:06:48.955621 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/minicluster-data/master-0/wal/instance
I20240507 02:06:48.955787 32269 external_mini_cluster.cc:1428] Setting key 767505ea3d5b0b6a477c976a0df68e2a
I20240507 02:06:48.955946 966 raft_consensus.cc:504] T 00000000000000000000000000000000 P c10b07de71ff43efb4d7a1d211569a29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c10b07de71ff43efb4d7a1d211569a29" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 44459 } }
I20240507 02:06:48.956033 966 leader_election.cc:302] T 00000000000000000000000000000000 P c10b07de71ff43efb4d7a1d211569a29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c10b07de71ff43efb4d7a1d211569a29; no voters:
I20240507 02:06:48.956159 966 leader_election.cc:288] T 00000000000000000000000000000000 P c10b07de71ff43efb4d7a1d211569a29 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240507 02:06:48.956321 969 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c10b07de71ff43efb4d7a1d211569a29 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:06:48.956378 966 sys_catalog.cc:564] T 00000000000000000000000000000000 P c10b07de71ff43efb4d7a1d211569a29 [sys.catalog]: configured and running, proceeding with master startup.
I20240507 02:06:48.956452 969 raft_consensus.cc:686] T 00000000000000000000000000000000 P c10b07de71ff43efb4d7a1d211569a29 [term 1 LEADER]: Becoming Leader. State: Replica: c10b07de71ff43efb4d7a1d211569a29, State: Running, Role: LEADER
I20240507 02:06:48.956529 969 consensus_queue.cc:237] T 00000000000000000000000000000000 P c10b07de71ff43efb4d7a1d211569a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c10b07de71ff43efb4d7a1d211569a29" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 44459 } }
I20240507 02:06:48.956801 969 sys_catalog.cc:455] T 00000000000000000000000000000000 P c10b07de71ff43efb4d7a1d211569a29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c10b07de71ff43efb4d7a1d211569a29. Latest consensus state: current_term: 1 leader_uuid: "c10b07de71ff43efb4d7a1d211569a29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c10b07de71ff43efb4d7a1d211569a29" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 44459 } } }
I20240507 02:06:48.956836 971 sys_catalog.cc:455] T 00000000000000000000000000000000 P c10b07de71ff43efb4d7a1d211569a29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c10b07de71ff43efb4d7a1d211569a29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c10b07de71ff43efb4d7a1d211569a29" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 44459 } } }
I20240507 02:06:48.956880 969 sys_catalog.cc:458] T 00000000000000000000000000000000 P c10b07de71ff43efb4d7a1d211569a29 [sys.catalog]: This master's current role is: LEADER
I20240507 02:06:48.956882 971 sys_catalog.cc:458] T 00000000000000000000000000000000 P c10b07de71ff43efb4d7a1d211569a29 [sys.catalog]: This master's current role is: LEADER
I20240507 02:06:48.957459 980 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 02:06:48.957867 980 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240507 02:06:48.959095 980 catalog_manager.cc:1347] Generated new cluster ID: a624b197ded641c9a9b6f4ece3b04e75
I20240507 02:06:48.959149 980 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
May 07 02:06:48 dist-test-slave-v0bg krb5kdc[874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047608, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:06:48.965365 972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:48.956192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55940 (local address 127.31.131.126:44459)
0507 02:06:48.956359 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:48.956371 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:48.957169 (+ 798us) server_negotiation.cc:407] Connection header received
0507 02:06:48.957306 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:48.957312 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:48.957361 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:48.958368 (+ 1007us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:06:48.958492 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:48.959564 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:48.960106 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:48.960305 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:48.962221 (+ 1916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:06:48.962227 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:06:48.962227 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:06:48.962243 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:06:48.964109 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:48.964423 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:48.964426 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:48.964426 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:48.964485 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:48.964611 (+ 126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:48.964612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:48.964612 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:48.964827 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:06:48.964955 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:48.965047 (+ 92us) server_negotiation.cc:299] Negotiation successful
0507 02:06:48.965172 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":79,"threads_started":1}
I20240507 02:06:48.984702 980 catalog_manager.cc:1370] Generated new certificate authority record
I20240507 02:06:48.985448 980 catalog_manager.cc:1504] Loading token signing keys...
I20240507 02:06:48.999024 980 catalog_manager.cc:5857] T 00000000000000000000000000000000 P c10b07de71ff43efb4d7a1d211569a29: Generated new TSK 0
I20240507 02:06:48.999213 980 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.131.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab.
Entry for principal kudu/127.31.131.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab.
I20240507 02:06:49.029227 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.65:0
--local_ip_for_outbound_sockets=127.31.131.65
--webserver_interface=127.31.131.65
--webserver_port=0
--tserver_master_addrs=127.31.131.126:44459
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab
--principal=kudu/127.31.131.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:45031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:06:49.103582 993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:06:49.103754 993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:06:49.103785 993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:06:49.104984 993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:06:49.105022 993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:06:49.105046 993 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20240507 02:06:49.105067 993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.65
I20240507 02:06:49.106577 993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:45031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab
--principal=kudu/127.31.131.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.131.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.131.126:44459
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.65
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:06:49.106781 993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:06:49.106945 993 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:06:49.110458 998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:06:49.110458 1001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:06:49.110617 999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:06:49.110620 993 server_base.cc:1034] running on GCE node
I20240507 02:06:49.110826 993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:06:49.110996 993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:06:49.112114 993 hybrid_clock.cc:648] HybridClock initialized: now 1715047609112102 us; error 26 us; skew 500 ppm
May 07 02:06:49 dist-test-slave-v0bg krb5kdc[874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047609, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:06:49.114192 993 init.cc:377] Logged in from keytab as kudu/127.31.131.65@KRBTEST.COM (short username kudu)
I20240507 02:06:49.114526 993 webserver.cc:460] Webserver started at http://127.31.131.65:46575/ using document root <none> and password file <none>
I20240507 02:06:49.114706 993 fs_manager.cc:359] Metadata directory not provided
I20240507 02:06:49.114743 993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:06:49.114836 993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:06:49.115720 993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/minicluster-data/ts-0/data/instance:
uuid: "549ef01dece445ddbd79862392dd53fd"
format_stamp: "Formatted at 2024-05-07 02:06:49 on dist-test-slave-v0bg"
server_key: "f5b273f948a6ec9e798679bb26ede911"
server_key_iv: "744dd141f1b3f406dd83fd2c17a0b48d"
server_key_version: "encryptionkey@0"
I20240507 02:06:49.115983 993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/minicluster-data/ts-0/wal/instance:
uuid: "549ef01dece445ddbd79862392dd53fd"
format_stamp: "Formatted at 2024-05-07 02:06:49 on dist-test-slave-v0bg"
server_key: "f5b273f948a6ec9e798679bb26ede911"
server_key_iv: "744dd141f1b3f406dd83fd2c17a0b48d"
server_key_version: "encryptionkey@0"
I20240507 02:06:49.116993 993 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240507 02:06:49.117539 1008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:49.117671 993 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240507 02:06:49.117736 993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/minicluster-data/ts-0/wal
uuid: "549ef01dece445ddbd79862392dd53fd"
format_stamp: "Formatted at 2024-05-07 02:06:49 on dist-test-slave-v0bg"
server_key: "f5b273f948a6ec9e798679bb26ede911"
server_key_iv: "744dd141f1b3f406dd83fd2c17a0b48d"
server_key_version: "encryptionkey@0"
I20240507 02:06:49.117790 993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:06:49.125536 993 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:06:49.126930 993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:06:49.127025 993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:06:49.127409 993 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:06:49.127444 993 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:49.127468 993 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:06:49.127485 993 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:49.132591 993 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.65:43451
I20240507 02:06:49.132668 1124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.65:43451 every 8 connection(s)
I20240507 02:06:49.132905 993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
May 07 02:06:49 dist-test-slave-v0bg krb5kdc[874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047609, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.65@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:06:49.136368 972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:49.127772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:54665 (local address 127.31.131.126:44459)
0507 02:06:49.127820 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:49.127823 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:49.127981 (+ 158us) server_negotiation.cc:407] Connection header received
0507 02:06:49.128501 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:49.128503 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:49.128517 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:49.128586 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:06:49.129500 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:49.130250 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:49.131035 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:49.131226 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:49.133995 (+ 2769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:06:49.134000 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:06:49.134000 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:06:49.134005 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:06:49.135320 (+ 1315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:49.135647 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:49.135648 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:49.135648 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:49.135670 (+ 22us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:49.135820 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:49.135821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:49.135821 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:49.135905 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:06:49.135959 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:49.136180 (+ 221us) server_negotiation.cc:299] Negotiation successful
0507 02:06:49.136271 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20240507 02:06:49.136435 1017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:49.127769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:44459 (local address 127.31.131.65:54665)
0507 02:06:49.127891 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0507 02:06:49.127913 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0507 02:06:49.128378 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:06:49.128646 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:06:49.128650 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:06:49.129017 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:06:49.129401 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:49.129405 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:49.130341 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:06:49.130342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:06:49.130941 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:49.130943 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:49.131072 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:49.131578 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:06:49.131619 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:06:49.133891 (+ 2272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:06:49.135395 (+ 1504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:06:49.135401 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:06:49.135403 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:06:49.135582 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:06:49.135720 (+ 138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:06:49.135721 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:06:49.135722 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:06:49.135772 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:06:49.135955 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:06:49.135960 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:06:49.136061 (+ 101us) client_negotiation.cc:769] Sending connection context
0507 02:06:49.136155 (+ 94us) client_negotiation.cc:240] Negotiation successful
0507 02:06:49.136272 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":72,"threads_started":1}
I20240507 02:06:49.136833 1125 heartbeater.cc:347] Connected to a master server at 127.31.131.126:44459
I20240507 02:06:49.136914 1125 heartbeater.cc:464] Registering TS with master...
I20240507 02:06:49.137112 1125 heartbeater.cc:511] Master 127.31.131.126:44459 requested a full tablet report, sending...
I20240507 02:06:49.137620 918 ts_manager.cc:194] Registered new tserver with Master: 549ef01dece445ddbd79862392dd53fd (127.31.131.65:43451)
I20240507 02:06:49.138334 918 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.131.65@KRBTEST.COM'} at 127.31.131.65:54665
I20240507 02:06:49.141937 972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:49.137940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:40985 (local address 127.31.131.126:44459)
0507 02:06:49.137983 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:49.137984 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:49.138086 (+ 102us) server_negotiation.cc:407] Connection header received
0507 02:06:49.138148 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:49.138149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:49.138158 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:49.138207 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:06:49.138703 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:49.139133 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:49.139624 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:49.139722 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:49.140259 (+ 537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:06:49.140261 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:06:49.140262 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:06:49.140265 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:06:49.141310 (+ 1045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:49.141541 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:49.141542 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:49.141542 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:49.141555 (+ 13us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:49.141646 (+ 91us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:49.141646 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:49.141647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:49.141710 (+ 63us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:06:49.141748 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:49.141833 (+ 85us) server_negotiation.cc:299] Negotiation successful
0507 02:06:49.141884 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20240507 02:06:49.141952 1131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:49.137891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:44459 (local address 127.31.131.65:40985)
0507 02:06:49.138023 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0507 02:06:49.138033 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0507 02:06:49.138088 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:06:49.138211 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:06:49.138212 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:06:49.138342 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:06:49.138643 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:49.138645 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:49.139199 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:06:49.139199 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:06:49.139567 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:49.139568 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:49.139621 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:49.139994 (+ 373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:06:49.139998 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:06:49.140206 (+ 208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:06:49.141372 (+ 1166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:06:49.141373 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:06:49.141373 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:06:49.141491 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:06:49.141596 (+ 105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:06:49.141596 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:06:49.141596 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:06:49.141616 (+ 20us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:06:49.141747 (+ 131us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:06:49.141748 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:06:49.141776 (+ 28us) client_negotiation.cc:769] Sending connection context
0507 02:06:49.141819 (+ 43us) client_negotiation.cc:240] Negotiation successful
0507 02:06:49.141879 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":87,"threads_started":1}
I20240507 02:06:49.142976 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 993
I20240507 02:06:49.143074 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/minicluster-data/ts-0/wal/instance
I20240507 02:06:49.143251 32269 external_mini_cluster.cc:1428] Setting key df9859d3628cc6b453ac53910cc7c33b
WARNING: no policy specified for kudu/127.31.131.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab.
Entry for principal kudu/127.31.131.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab.
I20240507 02:06:49.163228 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.66:0
--local_ip_for_outbound_sockets=127.31.131.66
--webserver_interface=127.31.131.66
--webserver_port=0
--tserver_master_addrs=127.31.131.126:44459
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab
--principal=kudu/127.31.131.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:45031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:06:49.237777 1134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:06:49.237926 1134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:06:49.237946 1134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:06:49.239166 1134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:06:49.239204 1134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:06:49.239223 1134 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20240507 02:06:49.239236 1134 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.66
I20240507 02:06:49.240706 1134 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:45031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab
--principal=kudu/127.31.131.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.131.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.131.126:44459
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1134
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.66
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:06:49.240860 1134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:06:49.240995 1134 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:06:49.244493 1142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:06:49.244530 1139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:06:49.244626 1140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:06:49.244611 1134 server_base.cc:1034] running on GCE node
I20240507 02:06:49.244859 1134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:06:49.245044 1134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:06:49.246174 1134 hybrid_clock.cc:648] HybridClock initialized: now 1715047609246157 us; error 29 us; skew 500 ppm
May 07 02:06:49 dist-test-slave-v0bg krb5kdc[874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047609, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:06:49.248239 1134 init.cc:377] Logged in from keytab as kudu/127.31.131.66@KRBTEST.COM (short username kudu)
I20240507 02:06:49.248576 1134 webserver.cc:460] Webserver started at http://127.31.131.66:36685/ using document root <none> and password file <none>
I20240507 02:06:49.248731 1134 fs_manager.cc:359] Metadata directory not provided
I20240507 02:06:49.248769 1134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:06:49.248863 1134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:06:49.249733 1134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/minicluster-data/ts-1/data/instance:
uuid: "79f1aff9a44840d6959b5ac8e4d072e9"
format_stamp: "Formatted at 2024-05-07 02:06:49 on dist-test-slave-v0bg"
server_key: "5522098022ef268b80f6f9e139002c1d"
server_key_iv: "d0fae736545ddad9723f0fed437de4e9"
server_key_version: "encryptionkey@0"
I20240507 02:06:49.250002 1134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/minicluster-data/ts-1/wal/instance:
uuid: "79f1aff9a44840d6959b5ac8e4d072e9"
format_stamp: "Formatted at 2024-05-07 02:06:49 on dist-test-slave-v0bg"
server_key: "5522098022ef268b80f6f9e139002c1d"
server_key_iv: "d0fae736545ddad9723f0fed437de4e9"
server_key_version: "encryptionkey@0"
I20240507 02:06:49.251073 1134 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240507 02:06:49.251650 1149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:49.251785 1134 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240507 02:06:49.251850 1134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/minicluster-data/ts-1/wal
uuid: "79f1aff9a44840d6959b5ac8e4d072e9"
format_stamp: "Formatted at 2024-05-07 02:06:49 on dist-test-slave-v0bg"
server_key: "5522098022ef268b80f6f9e139002c1d"
server_key_iv: "d0fae736545ddad9723f0fed437de4e9"
server_key_version: "encryptionkey@0"
I20240507 02:06:49.251904 1134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:06:49.266295 1134 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:06:49.267797 1134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:06:49.267894 1134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:06:49.268298 1134 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:06:49.268332 1134 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:49.268358 1134 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:06:49.268373 1134 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:49.274066 1134 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.66:34747
May 07 02:06:49 dist-test-slave-v0bg krb5kdc[874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047609, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.66@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:06:49.274243 1266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.66:34747 every 8 connection(s)
I20240507 02:06:49.274399 1134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
I20240507 02:06:49.276651 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 1134
I20240507 02:06:49.276736 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/minicluster-data/ts-1/wal/instance
I20240507 02:06:49.276885 32269 external_mini_cluster.cc:1428] Setting key 7f0823aa08c50ca1aadcd3cb132a0637
I20240507 02:06:49.276953 972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:49.268630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.66:57995 (local address 127.31.131.126:44459)
0507 02:06:49.268684 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:49.268686 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:49.268862 (+ 176us) server_negotiation.cc:407] Connection header received
0507 02:06:49.269366 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:49.269368 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:49.269380 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:49.269433 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:06:49.270368 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:49.271126 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:49.271857 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:49.272032 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:49.274769 (+ 2737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:06:49.274774 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:06:49.274774 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:06:49.274781 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:06:49.276018 (+ 1237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:49.276297 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:49.276298 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:49.276298 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:49.276314 (+ 16us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:49.276432 (+ 118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:49.276432 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:49.276432 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:49.276521 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:06:49.276577 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:49.276785 (+ 208us) server_negotiation.cc:299] Negotiation successful
0507 02:06:49.276861 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20240507 02:06:49.277025 1158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:49.268621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:44459 (local address 127.31.131.66:57995)
0507 02:06:49.268769 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0507 02:06:49.268791 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0507 02:06:49.269254 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:06:49.269515 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:06:49.269520 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:06:49.269899 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:06:49.270271 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:49.270277 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:49.271217 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:06:49.271218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:06:49.271765 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:49.271768 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:49.271913 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:49.272416 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:06:49.272436 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:06:49.274688 (+ 2252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:06:49.276081 (+ 1393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:06:49.276083 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:06:49.276085 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:06:49.276244 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:06:49.276358 (+ 114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:06:49.276358 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:06:49.276359 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:06:49.276401 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:06:49.276568 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:06:49.276570 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:06:49.276655 (+ 85us) client_negotiation.cc:769] Sending connection context
0507 02:06:49.276744 (+ 89us) client_negotiation.cc:240] Negotiation successful
0507 02:06:49.276863 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":88,"threads_started":1}
I20240507 02:06:49.277462 1267 heartbeater.cc:347] Connected to a master server at 127.31.131.126:44459
I20240507 02:06:49.277534 1267 heartbeater.cc:464] Registering TS with master...
I20240507 02:06:49.277724 1267 heartbeater.cc:511] Master 127.31.131.126:44459 requested a full tablet report, sending...
I20240507 02:06:49.278739 918 ts_manager.cc:194] Registered new tserver with Master: 79f1aff9a44840d6959b5ac8e4d072e9 (127.31.131.66:34747)
I20240507 02:06:49.279208 918 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.131.66@KRBTEST.COM'} at 127.31.131.66:57995
I20240507 02:06:49.282989 972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:49.278607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.66:44485 (local address 127.31.131.126:44459)
0507 02:06:49.278642 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:49.278644 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:49.278651 (+ 7us) server_negotiation.cc:407] Connection header received
0507 02:06:49.278668 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:49.278669 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:49.278678 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:49.278719 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:06:49.279145 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:49.279681 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:49.280310 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:49.280421 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:49.281026 (+ 605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:06:49.281028 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:06:49.281029 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:06:49.281034 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:06:49.282152 (+ 1118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:49.282449 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:49.282450 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:49.282450 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:49.282464 (+ 14us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:49.282602 (+ 138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:49.282602 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:49.282602 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:49.282699 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:06:49.282744 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:49.282878 (+ 134us) server_negotiation.cc:299] Negotiation successful
0507 02:06:49.282933 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20240507 02:06:49.283272 1273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:49.278353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:44459 (local address 127.31.131.66:44485)
0507 02:06:49.278463 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0507 02:06:49.278473 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0507 02:06:49.278523 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:06:49.278761 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:06:49.278762 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:06:49.278914 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:06:49.279076 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:49.279077 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:49.279742 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:06:49.279742 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:06:49.280234 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:49.280236 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:49.280303 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:49.280673 (+ 370us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:06:49.280677 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:06:49.280956 (+ 279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:06:49.282228 (+ 1272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:06:49.282229 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:06:49.282230 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:06:49.282384 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:06:49.282511 (+ 127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:06:49.282512 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:06:49.282512 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:06:49.282540 (+ 28us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:06:49.282752 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:06:49.282753 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:06:49.282793 (+ 40us) client_negotiation.cc:769] Sending connection context
0507 02:06:49.282859 (+ 66us) client_negotiation.cc:240] Negotiation successful
0507 02:06:49.282950 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":57,"threads_started":1}
WARNING: no policy specified for kudu/127.31.131.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab.
Entry for principal kudu/127.31.131.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab.
I20240507 02:06:49.298516 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.67:0
--local_ip_for_outbound_sockets=127.31.131.67
--webserver_interface=127.31.131.67
--webserver_port=0
--tserver_master_addrs=127.31.131.126:44459
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab
--principal=kudu/127.31.131.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:45031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:06:49.373288 1276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:06:49.373456 1276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:06:49.373476 1276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:06:49.374683 1276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:06:49.374717 1276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:06:49.374734 1276 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20240507 02:06:49.374754 1276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.67
I20240507 02:06:49.376227 1276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:45031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab
--principal=kudu/127.31.131.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.131.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.131.126:44459
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.67
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:06:49.376381 1276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:06:49.376514 1276 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240507 02:06:49.380163 1276 server_base.cc:1034] running on GCE node
W20240507 02:06:49.380211 1284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:06:49.380260 1282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:06:49.380345 1281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:06:49.380502 1276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:06:49.380671 1276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:06:49.381798 1276 hybrid_clock.cc:648] HybridClock initialized: now 1715047609381784 us; error 26 us; skew 500 ppm
May 07 02:06:49 dist-test-slave-v0bg krb5kdc[874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047609, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:06:49.383893 1276 init.cc:377] Logged in from keytab as kudu/127.31.131.67@KRBTEST.COM (short username kudu)
I20240507 02:06:49.384212 1276 webserver.cc:460] Webserver started at http://127.31.131.67:38915/ using document root <none> and password file <none>
I20240507 02:06:49.384347 1276 fs_manager.cc:359] Metadata directory not provided
I20240507 02:06:49.384379 1276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:06:49.384454 1276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:06:49.385299 1276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/minicluster-data/ts-2/data/instance:
uuid: "8bda3fbdf6ef4f66aee8ae42d035bb7e"
format_stamp: "Formatted at 2024-05-07 02:06:49 on dist-test-slave-v0bg"
server_key: "044dfce45608a039b551821b3e88f73f"
server_key_iv: "3f54445c9fbc55cd56d7b81026a6a153"
server_key_version: "encryptionkey@0"
I20240507 02:06:49.385541 1276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/minicluster-data/ts-2/wal/instance:
uuid: "8bda3fbdf6ef4f66aee8ae42d035bb7e"
format_stamp: "Formatted at 2024-05-07 02:06:49 on dist-test-slave-v0bg"
server_key: "044dfce45608a039b551821b3e88f73f"
server_key_iv: "3f54445c9fbc55cd56d7b81026a6a153"
server_key_version: "encryptionkey@0"
I20240507 02:06:49.386510 1276 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240507 02:06:49.387102 1291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:49.387261 1276 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:49.387326 1276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/minicluster-data/ts-2/wal
uuid: "8bda3fbdf6ef4f66aee8ae42d035bb7e"
format_stamp: "Formatted at 2024-05-07 02:06:49 on dist-test-slave-v0bg"
server_key: "044dfce45608a039b551821b3e88f73f"
server_key_iv: "3f54445c9fbc55cd56d7b81026a6a153"
server_key_version: "encryptionkey@0"
I20240507 02:06:49.387372 1276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:06:49.404289 1276 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:06:49.406045 1276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:06:49.406128 1276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:06:49.406533 1276 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:06:49.406601 1276 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:49.406623 1276 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:06:49.406643 1276 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:49.411964 1276 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.67:45181
I20240507 02:06:49.412015 1407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.67:45181 every 8 connection(s)
May 07 02:06:49 dist-test-slave-v0bg krb5kdc[874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047609, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.67@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:06:49.412263 1276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
I20240507 02:06:49.414686 972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:49.406974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.67:47059 (local address 127.31.131.126:44459)
0507 02:06:49.407039 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:49.407041 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:49.407173 (+ 132us) server_negotiation.cc:407] Connection header received
0507 02:06:49.407708 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:49.407710 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:49.407722 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:49.407804 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:06:49.408736 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:49.409456 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:49.410162 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:49.410326 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:49.412769 (+ 2443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:06:49.412773 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:06:49.412773 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:06:49.412780 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:06:49.413818 (+ 1038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:49.414110 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:49.414111 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:49.414111 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:49.414126 (+ 15us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:49.414246 (+ 120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:49.414246 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:49.414246 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:49.414318 (+ 72us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:06:49.414358 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:49.414532 (+ 174us) server_negotiation.cc:299] Negotiation successful
0507 02:06:49.414612 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20240507 02:06:49.414788 1300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:49.406927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:44459 (local address 127.31.131.67:47059)
0507 02:06:49.407068 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0507 02:06:49.407094 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0507 02:06:49.407585 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:06:49.407870 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:06:49.407874 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:06:49.408243 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:06:49.408638 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:49.408643 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:49.409539 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:06:49.409540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:06:49.410079 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:49.410080 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:49.410220 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:49.410724 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:06:49.410737 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:06:49.412686 (+ 1949us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:06:49.413879 (+ 1193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:06:49.413884 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:06:49.413886 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:06:49.414057 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:06:49.414167 (+ 110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:06:49.414167 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:06:49.414167 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:06:49.414210 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:06:49.414357 (+ 147us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:06:49.414359 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:06:49.414444 (+ 85us) client_negotiation.cc:769] Sending connection context
0507 02:06:49.414518 (+ 74us) client_negotiation.cc:240] Negotiation successful
0507 02:06:49.414643 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":79,"threads_started":1}
I20240507 02:06:49.415225 1408 heartbeater.cc:347] Connected to a master server at 127.31.131.126:44459
I20240507 02:06:49.415292 1408 heartbeater.cc:464] Registering TS with master...
I20240507 02:06:49.415452 1408 heartbeater.cc:511] Master 127.31.131.126:44459 requested a full tablet report, sending...
I20240507 02:06:49.415799 919 ts_manager.cc:194] Registered new tserver with Master: 8bda3fbdf6ef4f66aee8ae42d035bb7e (127.31.131.67:45181)
I20240507 02:06:49.416273 919 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.131.67@KRBTEST.COM'} at 127.31.131.67:47059
I20240507 02:06:49.420183 972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:49.416154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.67:45279 (local address 127.31.131.126:44459)
0507 02:06:49.416190 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:49.416192 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:49.416276 (+ 84us) server_negotiation.cc:407] Connection header received
0507 02:06:49.416343 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:49.416344 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:49.416352 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:49.416400 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:06:49.416830 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:49.417264 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:49.417728 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:49.417876 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:49.418400 (+ 524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:06:49.418403 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:06:49.418403 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:06:49.418409 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:06:49.419404 (+ 995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:49.419647 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:49.419648 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:49.419649 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:49.419666 (+ 17us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:49.419788 (+ 122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:49.419789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:49.419790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:49.419861 (+ 71us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:06:49.419909 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:49.420032 (+ 123us) server_negotiation.cc:299] Negotiation successful
0507 02:06:49.420091 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20240507 02:06:49.420192 1414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:49.416092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:44459 (local address 127.31.131.67:45279)
0507 02:06:49.416220 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0507 02:06:49.416229 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0507 02:06:49.416289 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:06:49.416445 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:06:49.416446 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:06:49.416592 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:06:49.416759 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:49.416761 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:49.417325 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:06:49.417326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:06:49.417655 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:49.417657 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:49.417716 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:49.418095 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:06:49.418099 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:06:49.418310 (+ 211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:06:49.419469 (+ 1159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:06:49.419470 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:06:49.419470 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:06:49.419593 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:06:49.419714 (+ 121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:06:49.419714 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:06:49.419715 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:06:49.419741 (+ 26us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:06:49.419907 (+ 166us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:06:49.419908 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:06:49.419945 (+ 37us) client_negotiation.cc:769] Sending connection context
0507 02:06:49.420004 (+ 59us) client_negotiation.cc:240] Negotiation successful
0507 02:06:49.420087 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":65,"threads_started":1}
I20240507 02:06:49.421860 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 1276
I20240507 02:06:49.421932 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/minicluster-data/ts-2/wal/instance
I20240507 02:06:49.422053 32269 external_mini_cluster.cc:1428] Setting key 2e67d6ce7c228a139f7ba83114a2dd15
I20240507 02:06:49.422783 32269 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
May 07 02:06:49 dist-test-slave-v0bg krb5kdc[874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047609, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
May 07 02:06:49 dist-test-slave-v0bg krb5kdc[874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047609, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:06:49.440693 972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:49.435359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55950 (local address 127.31.131.126:44459)
0507 02:06:49.435414 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:49.435415 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:49.435423 (+ 8us) server_negotiation.cc:407] Connection header received
0507 02:06:49.435470 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:49.435470 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:49.435478 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:49.435514 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:06:49.436101 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:49.436583 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:49.437096 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:49.437198 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:49.438950 (+ 1752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:06:49.438954 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:06:49.438954 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:06:49.438960 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:06:49.440083 (+ 1123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:49.440290 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:49.440291 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:49.440291 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:49.440306 (+ 15us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:49.440391 (+ 85us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:49.440391 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:49.440391 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:49.440459 (+ 68us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:06:49.440522 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:49.440583 (+ 61us) server_negotiation.cc:299] Negotiation successful
0507 02:06:49.440632 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20240507 02:06:49.442024 919 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55950:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 02:06:49.443495 919 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 02:06:49.448272 1426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:49.445508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.66:34747 (local address 127.0.0.1:57786)
0507 02:06:49.445610 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0507 02:06:49.445619 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0507 02:06:49.445673 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:06:49.445941 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:06:49.445942 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:06:49.445944 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:06:49.446126 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:49.446128 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:49.447077 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:06:49.447079 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:06:49.447760 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:49.447762 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:49.447829 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:49.447864 (+ 35us) client_negotiation.cc:769] Sending connection context
0507 02:06:49.447911 (+ 47us) client_negotiation.cc:240] Negotiation successful
0507 02:06:49.447944 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":70,"threads_started":1}
I20240507 02:06:49.448500 1422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:49.445199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.65:43451 (local address 127.0.0.1:36958)
0507 02:06:49.445302 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0507 02:06:49.445322 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0507 02:06:49.445383 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:06:49.445825 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:06:49.445825 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:06:49.445827 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:06:49.446083 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:49.446084 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:49.447511 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:06:49.447512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:06:49.448318 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:49.448321 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:49.448378 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:49.448381 (+ 3us) client_negotiation.cc:769] Sending connection context
0507 02:06:49.448409 (+ 28us) client_negotiation.cc:240] Negotiation successful
0507 02:06:49.448448 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79,"thread_start_us":46,"threads_started":1}
I20240507 02:06:49.448524 1427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:49.445602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57786 (local address 127.31.131.66:34747)
0507 02:06:49.445741 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:49.445747 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:49.445755 (+ 8us) server_negotiation.cc:407] Connection header received
0507 02:06:49.445785 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:49.445787 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:49.445799 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:49.445866 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:06:49.446197 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:49.447008 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:49.447860 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:49.448325 (+ 465us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:49.448378 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:49.448411 (+ 33us) server_negotiation.cc:299] Negotiation successful
0507 02:06:49.448451 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":72,"threads_started":1}
I20240507 02:06:49.448575 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:49.445386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.67:45181 (local address 127.0.0.1:42822)
0507 02:06:49.445469 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0507 02:06:49.445478 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0507 02:06:49.445523 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:06:49.445807 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:06:49.445809 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:06:49.445811 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:06:49.446082 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:49.446084 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:49.447555 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:06:49.447556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:06:49.448312 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:49.448314 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:49.448381 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:49.448385 (+ 4us) client_negotiation.cc:769] Sending connection context
0507 02:06:49.448412 (+ 27us) client_negotiation.cc:240] Negotiation successful
0507 02:06:49.448450 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69,"thread_start_us":38,"threads_started":1}
I20240507 02:06:49.449081 1425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:49.445473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42822 (local address 127.31.131.67:45181)
0507 02:06:49.445620 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:49.445627 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:49.445638 (+ 11us) server_negotiation.cc:407] Connection header received
0507 02:06:49.445687 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:49.445689 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:49.445706 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:49.445796 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:06:49.446269 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:49.447433 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:49.448549 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:49.448953 (+ 404us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:49.448982 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:49.449005 (+ 23us) server_negotiation.cc:299] Negotiation successful
0507 02:06:49.449031 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":73,"threads_started":1}
I20240507 02:06:49.449098 1200 tablet_service.cc:1467] Processing CreateTablet for tablet e21a16d27a494182a4e1face67521a27 (DEFAULT_TABLE table=test-table [id=451cd5d702de4b239915f55acd06ac94]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:06:49.449398 1200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e21a16d27a494182a4e1face67521a27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:06:49.449515 1342 tablet_service.cc:1467] Processing CreateTablet for tablet e21a16d27a494182a4e1face67521a27 (DEFAULT_TABLE table=test-table [id=451cd5d702de4b239915f55acd06ac94]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:06:49.449692 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:49.445298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36958 (local address 127.31.131.65:43451)
0507 02:06:49.445663 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:49.445671 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:49.445681 (+ 10us) server_negotiation.cc:407] Connection header received
0507 02:06:49.445720 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:49.445722 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:49.445737 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:49.446172 (+ 435us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:06:49.446254 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:49.447433 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:49.449044 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:49.449507 (+ 463us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:49.449546 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:49.449575 (+ 29us) server_negotiation.cc:299] Negotiation successful
0507 02:06:49.449613 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":83,"threads_started":1}
I20240507 02:06:49.449757 1342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e21a16d27a494182a4e1face67521a27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:06:49.450155 1058 tablet_service.cc:1467] Processing CreateTablet for tablet e21a16d27a494182a4e1face67521a27 (DEFAULT_TABLE table=test-table [id=451cd5d702de4b239915f55acd06ac94]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:06:49.450392 1058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e21a16d27a494182a4e1face67521a27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:06:49.451766 1428 tablet_bootstrap.cc:492] T e21a16d27a494182a4e1face67521a27 P 79f1aff9a44840d6959b5ac8e4d072e9: Bootstrap starting.
I20240507 02:06:49.452395 1428 tablet_bootstrap.cc:654] T e21a16d27a494182a4e1face67521a27 P 79f1aff9a44840d6959b5ac8e4d072e9: Neither blocks nor log segments found. Creating new log.
I20240507 02:06:49.452627 1429 tablet_bootstrap.cc:492] T e21a16d27a494182a4e1face67521a27 P 8bda3fbdf6ef4f66aee8ae42d035bb7e: Bootstrap starting.
I20240507 02:06:49.452777 1428 log.cc:824] T e21a16d27a494182a4e1face67521a27 P 79f1aff9a44840d6959b5ac8e4d072e9: Log is configured to *not* fsync() on all Append() calls
I20240507 02:06:49.453392 1429 tablet_bootstrap.cc:654] T e21a16d27a494182a4e1face67521a27 P 8bda3fbdf6ef4f66aee8ae42d035bb7e: Neither blocks nor log segments found. Creating new log.
I20240507 02:06:49.453495 1428 tablet_bootstrap.cc:492] T e21a16d27a494182a4e1face67521a27 P 79f1aff9a44840d6959b5ac8e4d072e9: No bootstrap required, opened a new log
I20240507 02:06:49.453527 1431 tablet_bootstrap.cc:492] T e21a16d27a494182a4e1face67521a27 P 549ef01dece445ddbd79862392dd53fd: Bootstrap starting.
I20240507 02:06:49.453634 1428 ts_tablet_manager.cc:1367] T e21a16d27a494182a4e1face67521a27 P 79f1aff9a44840d6959b5ac8e4d072e9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240507 02:06:49.453683 1429 log.cc:824] T e21a16d27a494182a4e1face67521a27 P 8bda3fbdf6ef4f66aee8ae42d035bb7e: Log is configured to *not* fsync() on all Append() calls
I20240507 02:06:49.454273 1429 tablet_bootstrap.cc:492] T e21a16d27a494182a4e1face67521a27 P 8bda3fbdf6ef4f66aee8ae42d035bb7e: No bootstrap required, opened a new log
I20240507 02:06:49.454283 1431 tablet_bootstrap.cc:654] T e21a16d27a494182a4e1face67521a27 P 549ef01dece445ddbd79862392dd53fd: Neither blocks nor log segments found. Creating new log.
I20240507 02:06:49.454329 1429 ts_tablet_manager.cc:1367] T e21a16d27a494182a4e1face67521a27 P 8bda3fbdf6ef4f66aee8ae42d035bb7e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240507 02:06:49.454538 1431 log.cc:824] T e21a16d27a494182a4e1face67521a27 P 549ef01dece445ddbd79862392dd53fd: Log is configured to *not* fsync() on all Append() calls
I20240507 02:06:49.455243 1431 tablet_bootstrap.cc:492] T e21a16d27a494182a4e1face67521a27 P 549ef01dece445ddbd79862392dd53fd: No bootstrap required, opened a new log
I20240507 02:06:49.455307 1431 ts_tablet_manager.cc:1367] T e21a16d27a494182a4e1face67521a27 P 549ef01dece445ddbd79862392dd53fd: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240507 02:06:49.455422 1428 raft_consensus.cc:348] T e21a16d27a494182a4e1face67521a27 P 79f1aff9a44840d6959b5ac8e4d072e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "549ef01dece445ddbd79862392dd53fd" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43451 } } peers { permanent_uuid: "8bda3fbdf6ef4f66aee8ae42d035bb7e" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 45181 } } peers { permanent_uuid: "79f1aff9a44840d6959b5ac8e4d072e9" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 34747 } }
I20240507 02:06:49.455583 1428 raft_consensus.cc:374] T e21a16d27a494182a4e1face67521a27 P 79f1aff9a44840d6959b5ac8e4d072e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:06:49.455606 1428 raft_consensus.cc:729] T e21a16d27a494182a4e1face67521a27 P 79f1aff9a44840d6959b5ac8e4d072e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79f1aff9a44840d6959b5ac8e4d072e9, State: Initialized, Role: FOLLOWER
I20240507 02:06:49.455690 1428 consensus_queue.cc:260] T e21a16d27a494182a4e1face67521a27 P 79f1aff9a44840d6959b5ac8e4d072e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "549ef01dece445ddbd79862392dd53fd" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43451 } } peers { permanent_uuid: "8bda3fbdf6ef4f66aee8ae42d035bb7e" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 45181 } } peers { permanent_uuid: "79f1aff9a44840d6959b5ac8e4d072e9" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 34747 } }
I20240507 02:06:49.455816 1429 raft_consensus.cc:348] T e21a16d27a494182a4e1face67521a27 P 8bda3fbdf6ef4f66aee8ae42d035bb7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "549ef01dece445ddbd79862392dd53fd" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43451 } } peers { permanent_uuid: "8bda3fbdf6ef4f66aee8ae42d035bb7e" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 45181 } } peers { permanent_uuid: "79f1aff9a44840d6959b5ac8e4d072e9" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 34747 } }
I20240507 02:06:49.455912 1428 ts_tablet_manager.cc:1398] T e21a16d27a494182a4e1face67521a27 P 79f1aff9a44840d6959b5ac8e4d072e9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240507 02:06:49.455936 1429 raft_consensus.cc:374] T e21a16d27a494182a4e1face67521a27 P 8bda3fbdf6ef4f66aee8ae42d035bb7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:06:49.455948 1267 heartbeater.cc:503] Master 127.31.131.126:44459 was elected leader, sending a full tablet report...
I20240507 02:06:49.455961 1429 raft_consensus.cc:729] T e21a16d27a494182a4e1face67521a27 P 8bda3fbdf6ef4f66aee8ae42d035bb7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bda3fbdf6ef4f66aee8ae42d035bb7e, State: Initialized, Role: FOLLOWER
I20240507 02:06:49.456046 1429 consensus_queue.cc:260] T e21a16d27a494182a4e1face67521a27 P 8bda3fbdf6ef4f66aee8ae42d035bb7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "549ef01dece445ddbd79862392dd53fd" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43451 } } peers { permanent_uuid: "8bda3fbdf6ef4f66aee8ae42d035bb7e" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 45181 } } peers { permanent_uuid: "79f1aff9a44840d6959b5ac8e4d072e9" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 34747 } }
I20240507 02:06:49.456266 1429 ts_tablet_manager.cc:1398] T e21a16d27a494182a4e1face67521a27 P 8bda3fbdf6ef4f66aee8ae42d035bb7e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240507 02:06:49.456359 1408 heartbeater.cc:503] Master 127.31.131.126:44459 was elected leader, sending a full tablet report...
I20240507 02:06:49.457273 1431 raft_consensus.cc:348] T e21a16d27a494182a4e1face67521a27 P 549ef01dece445ddbd79862392dd53fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "549ef01dece445ddbd79862392dd53fd" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43451 } } peers { permanent_uuid: "8bda3fbdf6ef4f66aee8ae42d035bb7e" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 45181 } } peers { permanent_uuid: "79f1aff9a44840d6959b5ac8e4d072e9" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 34747 } }
I20240507 02:06:49.457391 1431 raft_consensus.cc:374] T e21a16d27a494182a4e1face67521a27 P 549ef01dece445ddbd79862392dd53fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:06:49.457408 1431 raft_consensus.cc:729] T e21a16d27a494182a4e1face67521a27 P 549ef01dece445ddbd79862392dd53fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 549ef01dece445ddbd79862392dd53fd, State: Initialized, Role: FOLLOWER
I20240507 02:06:49.457486 1431 consensus_queue.cc:260] T e21a16d27a494182a4e1face67521a27 P 549ef01dece445ddbd79862392dd53fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "549ef01dece445ddbd79862392dd53fd" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43451 } } peers { permanent_uuid: "8bda3fbdf6ef4f66aee8ae42d035bb7e" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 45181 } } peers { permanent_uuid: "79f1aff9a44840d6959b5ac8e4d072e9" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 34747 } }
I20240507 02:06:49.457659 1431 ts_tablet_manager.cc:1398] T e21a16d27a494182a4e1face67521a27 P 549ef01dece445ddbd79862392dd53fd: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240507 02:06:49.457729 1125 heartbeater.cc:503] Master 127.31.131.126:44459 was elected leader, sending a full tablet report...
W20240507 02:06:49.524896 1268 tablet.cc:2369] T e21a16d27a494182a4e1face67521a27 P 79f1aff9a44840d6959b5ac8e4d072e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240507 02:06:49.584002 1436 raft_consensus.cc:482] T e21a16d27a494182a4e1face67521a27 P 549ef01dece445ddbd79862392dd53fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 02:06:49.584124 1436 raft_consensus.cc:504] T e21a16d27a494182a4e1face67521a27 P 549ef01dece445ddbd79862392dd53fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "549ef01dece445ddbd79862392dd53fd" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43451 } } peers { permanent_uuid: "8bda3fbdf6ef4f66aee8ae42d035bb7e" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 45181 } } peers { permanent_uuid: "79f1aff9a44840d6959b5ac8e4d072e9" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 34747 } }
I20240507 02:06:49.584443 1436 leader_election.cc:288] T e21a16d27a494182a4e1face67521a27 P 549ef01dece445ddbd79862392dd53fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8bda3fbdf6ef4f66aee8ae42d035bb7e (127.31.131.67:45181), 79f1aff9a44840d6959b5ac8e4d072e9 (127.31.131.66:34747)
I20240507 02:06:49.587219 1017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:49.584655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.67:45181 (local address 127.31.131.65:60661)
0507 02:06:49.584708 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0507 02:06:49.584718 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0507 02:06:49.584780 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:06:49.584940 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:06:49.584941 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:06:49.584943 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:06:49.585162 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:49.585165 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:49.586038 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:06:49.586039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:06:49.586945 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:49.586947 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:49.587036 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:49.587041 (+ 5us) client_negotiation.cc:769] Sending connection context
0507 02:06:49.587076 (+ 35us) client_negotiation.cc:240] Negotiation successful
0507 02:06:49.587125 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20240507 02:06:49.587230 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:49.584759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.66:34747 (local address 127.31.131.65:58393)
0507 02:06:49.584930 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0507 02:06:49.584938 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0507 02:06:49.584995 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:06:49.585104 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:06:49.585104 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:06:49.585106 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:06:49.585295 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:49.585297 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:49.586155 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:06:49.586156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:06:49.586991 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:49.586993 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:49.587058 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:49.587061 (+ 3us) client_negotiation.cc:769] Sending connection context
0507 02:06:49.587095 (+ 34us) client_negotiation.cc:240] Negotiation successful
0507 02:06:49.587125 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":111,"threads_started":1}
I20240507 02:06:49.587604 1425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:49.584720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:60661 (local address 127.31.131.67:45181)
0507 02:06:49.584777 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:49.584778 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:49.584787 (+ 9us) server_negotiation.cc:407] Connection header received
0507 02:06:49.584834 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:49.584835 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:49.584843 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:49.584882 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:06:49.585234 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:49.585938 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:49.587031 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:49.587483 (+ 452us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:49.587491 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:49.587510 (+ 19us) server_negotiation.cc:299] Negotiation successful
0507 02:06:49.587538 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20240507 02:06:49.587604 1427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:49.584850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:58393 (local address 127.31.131.66:34747)
0507 02:06:49.584911 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:49.584913 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:49.585004 (+ 91us) server_negotiation.cc:407] Connection header received
0507 02:06:49.585032 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:49.585033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:49.585041 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:49.585092 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:06:49.585359 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:49.586054 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:49.587056 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:49.587483 (+ 427us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:49.587491 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:49.587510 (+ 19us) server_negotiation.cc:299] Negotiation successful
0507 02:06:49.587538 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20240507 02:06:49.587888 1220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e21a16d27a494182a4e1face67521a27" candidate_uuid: "549ef01dece445ddbd79862392dd53fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79f1aff9a44840d6959b5ac8e4d072e9" is_pre_election: true
I20240507 02:06:49.587899 1362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e21a16d27a494182a4e1face67521a27" candidate_uuid: "549ef01dece445ddbd79862392dd53fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bda3fbdf6ef4f66aee8ae42d035bb7e" is_pre_election: true
I20240507 02:06:49.588009 1220 raft_consensus.cc:2454] T e21a16d27a494182a4e1face67521a27 P 79f1aff9a44840d6959b5ac8e4d072e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 549ef01dece445ddbd79862392dd53fd in term 0.
I20240507 02:06:49.588032 1362 raft_consensus.cc:2454] T e21a16d27a494182a4e1face67521a27 P 8bda3fbdf6ef4f66aee8ae42d035bb7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 549ef01dece445ddbd79862392dd53fd in term 0.
I20240507 02:06:49.588200 1012 leader_election.cc:302] T e21a16d27a494182a4e1face67521a27 P 549ef01dece445ddbd79862392dd53fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 549ef01dece445ddbd79862392dd53fd, 79f1aff9a44840d6959b5ac8e4d072e9; no voters:
I20240507 02:06:49.588326 1436 raft_consensus.cc:2781] T e21a16d27a494182a4e1face67521a27 P 549ef01dece445ddbd79862392dd53fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 02:06:49.588395 1436 raft_consensus.cc:482] T e21a16d27a494182a4e1face67521a27 P 549ef01dece445ddbd79862392dd53fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 02:06:49.588423 1436 raft_consensus.cc:3037] T e21a16d27a494182a4e1face67521a27 P 549ef01dece445ddbd79862392dd53fd [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:06:49.589028 1436 raft_consensus.cc:504] T e21a16d27a494182a4e1face67521a27 P 549ef01dece445ddbd79862392dd53fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "549ef01dece445ddbd79862392dd53fd" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43451 } } peers { permanent_uuid: "8bda3fbdf6ef4f66aee8ae42d035bb7e" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 45181 } } peers { permanent_uuid: "79f1aff9a44840d6959b5ac8e4d072e9" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 34747 } }
I20240507 02:06:49.589138 1436 leader_election.cc:288] T e21a16d27a494182a4e1face67521a27 P 549ef01dece445ddbd79862392dd53fd [CANDIDATE]: Term 1 election: Requested vote from peers 8bda3fbdf6ef4f66aee8ae42d035bb7e (127.31.131.67:45181), 79f1aff9a44840d6959b5ac8e4d072e9 (127.31.131.66:34747)
I20240507 02:06:49.589309 1362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e21a16d27a494182a4e1face67521a27" candidate_uuid: "549ef01dece445ddbd79862392dd53fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bda3fbdf6ef4f66aee8ae42d035bb7e"
I20240507 02:06:49.589339 1220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e21a16d27a494182a4e1face67521a27" candidate_uuid: "549ef01dece445ddbd79862392dd53fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79f1aff9a44840d6959b5ac8e4d072e9"
I20240507 02:06:49.589378 1362 raft_consensus.cc:3037] T e21a16d27a494182a4e1face67521a27 P 8bda3fbdf6ef4f66aee8ae42d035bb7e [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:06:49.589390 1220 raft_consensus.cc:3037] T e21a16d27a494182a4e1face67521a27 P 79f1aff9a44840d6959b5ac8e4d072e9 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:06:49.589999 1220 raft_consensus.cc:2454] T e21a16d27a494182a4e1face67521a27 P 79f1aff9a44840d6959b5ac8e4d072e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 549ef01dece445ddbd79862392dd53fd in term 1.
I20240507 02:06:49.590091 1362 raft_consensus.cc:2454] T e21a16d27a494182a4e1face67521a27 P 8bda3fbdf6ef4f66aee8ae42d035bb7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 549ef01dece445ddbd79862392dd53fd in term 1.
I20240507 02:06:49.590139 1012 leader_election.cc:302] T e21a16d27a494182a4e1face67521a27 P 549ef01dece445ddbd79862392dd53fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 549ef01dece445ddbd79862392dd53fd, 79f1aff9a44840d6959b5ac8e4d072e9; no voters:
I20240507 02:06:49.590243 1436 raft_consensus.cc:2781] T e21a16d27a494182a4e1face67521a27 P 549ef01dece445ddbd79862392dd53fd [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:06:49.590375 1436 raft_consensus.cc:686] T e21a16d27a494182a4e1face67521a27 P 549ef01dece445ddbd79862392dd53fd [term 1 LEADER]: Becoming Leader. State: Replica: 549ef01dece445ddbd79862392dd53fd, State: Running, Role: LEADER
I20240507 02:06:49.590448 1436 consensus_queue.cc:237] T e21a16d27a494182a4e1face67521a27 P 549ef01dece445ddbd79862392dd53fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "549ef01dece445ddbd79862392dd53fd" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43451 } } peers { permanent_uuid: "8bda3fbdf6ef4f66aee8ae42d035bb7e" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 45181 } } peers { permanent_uuid: "79f1aff9a44840d6959b5ac8e4d072e9" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 34747 } }
I20240507 02:06:49.591192 919 catalog_manager.cc:5484] T e21a16d27a494182a4e1face67521a27 P 549ef01dece445ddbd79862392dd53fd reported cstate change: term changed from 0 to 1, leader changed from <none> to 549ef01dece445ddbd79862392dd53fd (127.31.131.65). New cstate: current_term: 1 leader_uuid: "549ef01dece445ddbd79862392dd53fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "549ef01dece445ddbd79862392dd53fd" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8bda3fbdf6ef4f66aee8ae42d035bb7e" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 45181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79f1aff9a44840d6959b5ac8e4d072e9" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 34747 } health_report { overall_health: UNKNOWN } } }
I20240507 02:06:49.628466 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:49.626511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36962 (local address 127.31.131.65:43451)
0507 02:06:49.626594 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:49.626596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:49.626613 (+ 17us) server_negotiation.cc:407] Connection header received
0507 02:06:49.626696 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:49.626696 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:49.626713 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:49.626764 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0507 02:06:49.627008 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:49.627448 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:49.627995 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:49.628135 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:49.628170 (+ 35us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0507 02:06:49.628248 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0507 02:06:49.628309 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:49.628372 (+ 63us) server_negotiation.cc:299] Negotiation successful
0507 02:06:49.628394 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20240507 02:06:49.630012 1362 raft_consensus.cc:1261] T e21a16d27a494182a4e1face67521a27 P 8bda3fbdf6ef4f66aee8ae42d035bb7e [term 1 FOLLOWER]: Refusing update from remote peer 549ef01dece445ddbd79862392dd53fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 02:06:49.630012 1220 raft_consensus.cc:1261] T e21a16d27a494182a4e1face67521a27 P 79f1aff9a44840d6959b5ac8e4d072e9 [term 1 FOLLOWER]: Refusing update from remote peer 549ef01dece445ddbd79862392dd53fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 02:06:49.630347 1438 consensus_queue.cc:1035] T e21a16d27a494182a4e1face67521a27 P 549ef01dece445ddbd79862392dd53fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8bda3fbdf6ef4f66aee8ae42d035bb7e" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 45181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 02:06:49.630470 1436 consensus_queue.cc:1035] T e21a16d27a494182a4e1face67521a27 P 549ef01dece445ddbd79862392dd53fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "79f1aff9a44840d6959b5ac8e4d072e9" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 02:06:49.632655 1443 mvcc.cc:205] Tried to move back new op lower bound from 7024835009042067456 to 7024835008883036160. Current Snapshot: MvccSnapshot[applied={T|T < 7024835009042067456}]
I20240507 02:06:49.632938 1442 mvcc.cc:205] Tried to move back new op lower bound from 7024835009042067456 to 7024835008883036160. Current Snapshot: MvccSnapshot[applied={T|T < 7024835009042067456}]
I20240507 02:06:49.635496 918 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55950:
table { table_name: "test-table" } modify_external_catalogs: true
I20240507 02:06:49.635574 918 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55950:
table { table_name: "test-table" } modify_external_catalogs: true
I20240507 02:06:49.636219 918 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c10b07de71ff43efb4d7a1d211569a29: Sending DeleteTablet for 3 replicas of tablet e21a16d27a494182a4e1face67521a27
I20240507 02:06:49.636546 1342 tablet_service.cc:1514] Processing DeleteTablet for tablet e21a16d27a494182a4e1face67521a27 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:06:49 UTC) from {username='kudu', principal='kudu/127.31.131.126@KRBTEST.COM'} at 127.0.0.1:42822
I20240507 02:06:49.636554 1058 tablet_service.cc:1514] Processing DeleteTablet for tablet e21a16d27a494182a4e1face67521a27 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:06:49 UTC) from {username='kudu', principal='kudu/127.31.131.126@KRBTEST.COM'} at 127.0.0.1:36958
I20240507 02:06:49.636566 1200 tablet_service.cc:1514] Processing DeleteTablet for tablet e21a16d27a494182a4e1face67521a27 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:06:49 UTC) from {username='kudu', principal='kudu/127.31.131.126@KRBTEST.COM'} at 127.0.0.1:57786
I20240507 02:06:49.636804 1453 tablet_replica.cc:330] T e21a16d27a494182a4e1face67521a27 P 79f1aff9a44840d6959b5ac8e4d072e9: stopping tablet replica
I20240507 02:06:49.636888 1453 raft_consensus.cc:2229] T e21a16d27a494182a4e1face67521a27 P 79f1aff9a44840d6959b5ac8e4d072e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 02:06:49.636974 1453 raft_consensus.cc:2258] T e21a16d27a494182a4e1face67521a27 P 79f1aff9a44840d6959b5ac8e4d072e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 02:06:49.636969 1452 tablet_replica.cc:330] T e21a16d27a494182a4e1face67521a27 P 549ef01dece445ddbd79862392dd53fd: stopping tablet replica
I20240507 02:06:49.636952 918 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55950:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 02:06:49.637030 1452 raft_consensus.cc:2229] T e21a16d27a494182a4e1face67521a27 P 549ef01dece445ddbd79862392dd53fd [term 1 LEADER]: Raft consensus shutting down.
W20240507 02:06:49.637082 918 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 02:06:49.637099 1452 raft_consensus.cc:2258] T e21a16d27a494182a4e1face67521a27 P 549ef01dece445ddbd79862392dd53fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 02:06:49.637243 1451 tablet_replica.cc:330] T e21a16d27a494182a4e1face67521a27 P 8bda3fbdf6ef4f66aee8ae42d035bb7e: stopping tablet replica
I20240507 02:06:49.637306 1451 raft_consensus.cc:2229] T e21a16d27a494182a4e1face67521a27 P 8bda3fbdf6ef4f66aee8ae42d035bb7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 02:06:49.637373 1451 raft_consensus.cc:2258] T e21a16d27a494182a4e1face67521a27 P 8bda3fbdf6ef4f66aee8ae42d035bb7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 02:06:49.637493 1452 ts_tablet_manager.cc:1875] T e21a16d27a494182a4e1face67521a27 P 549ef01dece445ddbd79862392dd53fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 02:06:49.637818 1451 ts_tablet_manager.cc:1875] T e21a16d27a494182a4e1face67521a27 P 8bda3fbdf6ef4f66aee8ae42d035bb7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 02:06:49.638309 1453 ts_tablet_manager.cc:1875] T e21a16d27a494182a4e1face67521a27 P 79f1aff9a44840d6959b5ac8e4d072e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 02:06:49.638440 1058 tablet_service.cc:1467] Processing CreateTablet for tablet 8e698acfb3f848d2be86024d62ea5b43 (DEFAULT_TABLE table=test-table [id=35eac029cf304da982b723194ace138b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:06:49.638526 1058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e698acfb3f848d2be86024d62ea5b43. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:06:49.638624 1342 tablet_service.cc:1467] Processing CreateTablet for tablet 8e698acfb3f848d2be86024d62ea5b43 (DEFAULT_TABLE table=test-table [id=35eac029cf304da982b723194ace138b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:06:49.638723 1342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e698acfb3f848d2be86024d62ea5b43. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:06:49.639376 1451 ts_tablet_manager.cc:1888] T e21a16d27a494182a4e1face67521a27 P 8bda3fbdf6ef4f66aee8ae42d035bb7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240507 02:06:49.639427 1451 log.cc:1196] T e21a16d27a494182a4e1face67521a27 P 8bda3fbdf6ef4f66aee8ae42d035bb7e: Deleting WAL directory at /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/minicluster-data/ts-2/wal/wals/e21a16d27a494182a4e1face67521a27
I20240507 02:06:49.639659 1451 ts_tablet_manager.cc:1909] T e21a16d27a494182a4e1face67521a27 P 8bda3fbdf6ef4f66aee8ae42d035bb7e: Deleting consensus metadata
I20240507 02:06:49.639990 906 catalog_manager.cc:4830] TS 8bda3fbdf6ef4f66aee8ae42d035bb7e (127.31.131.67:45181): tablet e21a16d27a494182a4e1face67521a27 (table test-table [id=451cd5d702de4b239915f55acd06ac94]) successfully deleted
I20240507 02:06:49.639048 1452 ts_tablet_manager.cc:1888] T e21a16d27a494182a4e1face67521a27 P 549ef01dece445ddbd79862392dd53fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240507 02:06:49.640187 1429 tablet_bootstrap.cc:492] T 8e698acfb3f848d2be86024d62ea5b43 P 8bda3fbdf6ef4f66aee8ae42d035bb7e: Bootstrap starting.
I20240507 02:06:49.640194 1453 ts_tablet_manager.cc:1888] T e21a16d27a494182a4e1face67521a27 P 79f1aff9a44840d6959b5ac8e4d072e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240507 02:06:49.640225 1452 log.cc:1196] T e21a16d27a494182a4e1face67521a27 P 549ef01dece445ddbd79862392dd53fd: Deleting WAL directory at /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/minicluster-data/ts-0/wal/wals/e21a16d27a494182a4e1face67521a27
I20240507 02:06:49.640241 1453 log.cc:1196] T e21a16d27a494182a4e1face67521a27 P 79f1aff9a44840d6959b5ac8e4d072e9: Deleting WAL directory at /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/minicluster-data/ts-1/wal/wals/e21a16d27a494182a4e1face67521a27
I20240507 02:06:49.640455 1452 ts_tablet_manager.cc:1909] T e21a16d27a494182a4e1face67521a27 P 549ef01dece445ddbd79862392dd53fd: Deleting consensus metadata
I20240507 02:06:49.640525 1429 tablet_bootstrap.cc:654] T 8e698acfb3f848d2be86024d62ea5b43 P 8bda3fbdf6ef4f66aee8ae42d035bb7e: Neither blocks nor log segments found. Creating new log.
I20240507 02:06:49.640676 1453 ts_tablet_manager.cc:1909] T e21a16d27a494182a4e1face67521a27 P 79f1aff9a44840d6959b5ac8e4d072e9: Deleting consensus metadata
I20240507 02:06:49.641144 1429 tablet_bootstrap.cc:492] T 8e698acfb3f848d2be86024d62ea5b43 P 8bda3fbdf6ef4f66aee8ae42d035bb7e: No bootstrap required, opened a new log
I20240507 02:06:49.641180 1429 ts_tablet_manager.cc:1367] T 8e698acfb3f848d2be86024d62ea5b43 P 8bda3fbdf6ef4f66aee8ae42d035bb7e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240507 02:06:49.641290 1431 tablet_bootstrap.cc:492] T 8e698acfb3f848d2be86024d62ea5b43 P 549ef01dece445ddbd79862392dd53fd: Bootstrap starting.
I20240507 02:06:49.641289 1429 raft_consensus.cc:348] T 8e698acfb3f848d2be86024d62ea5b43 P 8bda3fbdf6ef4f66aee8ae42d035bb7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "549ef01dece445ddbd79862392dd53fd" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43451 } } peers { permanent_uuid: "79f1aff9a44840d6959b5ac8e4d072e9" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 34747 } } peers { permanent_uuid: "8bda3fbdf6ef4f66aee8ae42d035bb7e" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 45181 } }
I20240507 02:06:49.641355 1429 raft_consensus.cc:374] T 8e698acfb3f848d2be86024d62ea5b43 P 8bda3fbdf6ef4f66aee8ae42d035bb7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:06:49.641350 906 catalog_manager.cc:4830] TS 79f1aff9a44840d6959b5ac8e4d072e9 (127.31.131.66:34747): tablet e21a16d27a494182a4e1face67521a27 (table test-table [id=451cd5d702de4b239915f55acd06ac94]) successfully deleted
I20240507 02:06:49.641374 1429 raft_consensus.cc:729] T 8e698acfb3f848d2be86024d62ea5b43 P 8bda3fbdf6ef4f66aee8ae42d035bb7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bda3fbdf6ef4f66aee8ae42d035bb7e, State: Initialized, Role: FOLLOWER
I20240507 02:06:49.641417 1429 consensus_queue.cc:260] T 8e698acfb3f848d2be86024d62ea5b43 P 8bda3fbdf6ef4f66aee8ae42d035bb7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "549ef01dece445ddbd79862392dd53fd" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43451 } } peers { permanent_uuid: "79f1aff9a44840d6959b5ac8e4d072e9" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 34747 } } peers { permanent_uuid: "8bda3fbdf6ef4f66aee8ae42d035bb7e" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 45181 } }
I20240507 02:06:49.641489 1429 ts_tablet_manager.cc:1398] T 8e698acfb3f848d2be86024d62ea5b43 P 8bda3fbdf6ef4f66aee8ae42d035bb7e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:49.641486 906 catalog_manager.cc:4830] TS 549ef01dece445ddbd79862392dd53fd (127.31.131.65:43451): tablet e21a16d27a494182a4e1face67521a27 (table test-table [id=451cd5d702de4b239915f55acd06ac94]) successfully deleted
I20240507 02:06:49.641625 1431 tablet_bootstrap.cc:654] T 8e698acfb3f848d2be86024d62ea5b43 P 549ef01dece445ddbd79862392dd53fd: Neither blocks nor log segments found. Creating new log.
I20240507 02:06:49.641636 1200 tablet_service.cc:1467] Processing CreateTablet for tablet 8e698acfb3f848d2be86024d62ea5b43 (DEFAULT_TABLE table=test-table [id=35eac029cf304da982b723194ace138b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:06:49.641722 1200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e698acfb3f848d2be86024d62ea5b43. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:06:49.642305 1431 tablet_bootstrap.cc:492] T 8e698acfb3f848d2be86024d62ea5b43 P 549ef01dece445ddbd79862392dd53fd: No bootstrap required, opened a new log
I20240507 02:06:49.642342 1431 ts_tablet_manager.cc:1367] T 8e698acfb3f848d2be86024d62ea5b43 P 549ef01dece445ddbd79862392dd53fd: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20240507 02:06:49.642450 1431 raft_consensus.cc:348] T 8e698acfb3f848d2be86024d62ea5b43 P 549ef01dece445ddbd79862392dd53fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "549ef01dece445ddbd79862392dd53fd" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43451 } } peers { permanent_uuid: "79f1aff9a44840d6959b5ac8e4d072e9" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 34747 } } peers { permanent_uuid: "8bda3fbdf6ef4f66aee8ae42d035bb7e" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 45181 } }
I20240507 02:06:49.642489 1431 raft_consensus.cc:374] T 8e698acfb3f848d2be86024d62ea5b43 P 549ef01dece445ddbd79862392dd53fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:06:49.642500 1431 raft_consensus.cc:729] T 8e698acfb3f848d2be86024d62ea5b43 P 549ef01dece445ddbd79862392dd53fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 549ef01dece445ddbd79862392dd53fd, State: Initialized, Role: FOLLOWER
I20240507 02:06:49.642525 1431 consensus_queue.cc:260] T 8e698acfb3f848d2be86024d62ea5b43 P 549ef01dece445ddbd79862392dd53fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "549ef01dece445ddbd79862392dd53fd" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43451 } } peers { permanent_uuid: "79f1aff9a44840d6959b5ac8e4d072e9" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 34747 } } peers { permanent_uuid: "8bda3fbdf6ef4f66aee8ae42d035bb7e" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 45181 } }
I20240507 02:06:49.642627 1431 ts_tablet_manager.cc:1398] T 8e698acfb3f848d2be86024d62ea5b43 P 549ef01dece445ddbd79862392dd53fd: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:49.642794 1428 tablet_bootstrap.cc:492] T 8e698acfb3f848d2be86024d62ea5b43 P 79f1aff9a44840d6959b5ac8e4d072e9: Bootstrap starting.
I20240507 02:06:49.643035 1428 tablet_bootstrap.cc:654] T 8e698acfb3f848d2be86024d62ea5b43 P 79f1aff9a44840d6959b5ac8e4d072e9: Neither blocks nor log segments found. Creating new log.
I20240507 02:06:49.643620 1428 tablet_bootstrap.cc:492] T 8e698acfb3f848d2be86024d62ea5b43 P 79f1aff9a44840d6959b5ac8e4d072e9: No bootstrap required, opened a new log
I20240507 02:06:49.643672 1428 ts_tablet_manager.cc:1367] T 8e698acfb3f848d2be86024d62ea5b43 P 79f1aff9a44840d6959b5ac8e4d072e9: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20240507 02:06:49.643793 1428 raft_consensus.cc:348] T 8e698acfb3f848d2be86024d62ea5b43 P 79f1aff9a44840d6959b5ac8e4d072e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "549ef01dece445ddbd79862392dd53fd" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43451 } } peers { permanent_uuid: "79f1aff9a44840d6959b5ac8e4d072e9" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 34747 } } peers { permanent_uuid: "8bda3fbdf6ef4f66aee8ae42d035bb7e" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 45181 } }
I20240507 02:06:49.643838 1428 raft_consensus.cc:374] T 8e698acfb3f848d2be86024d62ea5b43 P 79f1aff9a44840d6959b5ac8e4d072e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:06:49.643849 1428 raft_consensus.cc:729] T 8e698acfb3f848d2be86024d62ea5b43 P 79f1aff9a44840d6959b5ac8e4d072e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79f1aff9a44840d6959b5ac8e4d072e9, State: Initialized, Role: FOLLOWER
I20240507 02:06:49.643879 1428 consensus_queue.cc:260] T 8e698acfb3f848d2be86024d62ea5b43 P 79f1aff9a44840d6959b5ac8e4d072e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "549ef01dece445ddbd79862392dd53fd" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43451 } } peers { permanent_uuid: "79f1aff9a44840d6959b5ac8e4d072e9" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 34747 } } peers { permanent_uuid: "8bda3fbdf6ef4f66aee8ae42d035bb7e" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 45181 } }
I20240507 02:06:49.643950 1428 ts_tablet_manager.cc:1398] T 8e698acfb3f848d2be86024d62ea5b43 P 79f1aff9a44840d6959b5ac8e4d072e9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20240507 02:06:49.662703 1409 tablet.cc:2369] T 8e698acfb3f848d2be86024d62ea5b43 P 8bda3fbdf6ef4f66aee8ae42d035bb7e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240507 02:06:49.689379 1438 raft_consensus.cc:482] T 8e698acfb3f848d2be86024d62ea5b43 P 549ef01dece445ddbd79862392dd53fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 02:06:49.689471 1438 raft_consensus.cc:504] T 8e698acfb3f848d2be86024d62ea5b43 P 549ef01dece445ddbd79862392dd53fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "549ef01dece445ddbd79862392dd53fd" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43451 } } peers { permanent_uuid: "79f1aff9a44840d6959b5ac8e4d072e9" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 34747 } } peers { permanent_uuid: "8bda3fbdf6ef4f66aee8ae42d035bb7e" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 45181 } }
I20240507 02:06:49.689625 1438 leader_election.cc:288] T 8e698acfb3f848d2be86024d62ea5b43 P 549ef01dece445ddbd79862392dd53fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79f1aff9a44840d6959b5ac8e4d072e9 (127.31.131.66:34747), 8bda3fbdf6ef4f66aee8ae42d035bb7e (127.31.131.67:45181)
I20240507 02:06:49.689989 1220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e698acfb3f848d2be86024d62ea5b43" candidate_uuid: "549ef01dece445ddbd79862392dd53fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79f1aff9a44840d6959b5ac8e4d072e9" is_pre_election: true
I20240507 02:06:49.690083 1220 raft_consensus.cc:2454] T 8e698acfb3f848d2be86024d62ea5b43 P 79f1aff9a44840d6959b5ac8e4d072e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 549ef01dece445ddbd79862392dd53fd in term 0.
I20240507 02:06:49.690241 1012 leader_election.cc:302] T 8e698acfb3f848d2be86024d62ea5b43 P 549ef01dece445ddbd79862392dd53fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 549ef01dece445ddbd79862392dd53fd, 79f1aff9a44840d6959b5ac8e4d072e9; no voters:
I20240507 02:06:49.690325 1438 raft_consensus.cc:2781] T 8e698acfb3f848d2be86024d62ea5b43 P 549ef01dece445ddbd79862392dd53fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 02:06:49.690347 1438 raft_consensus.cc:482] T 8e698acfb3f848d2be86024d62ea5b43 P 549ef01dece445ddbd79862392dd53fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 02:06:49.690366 1438 raft_consensus.cc:3037] T 8e698acfb3f848d2be86024d62ea5b43 P 549ef01dece445ddbd79862392dd53fd [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:06:49.690687 1362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e698acfb3f848d2be86024d62ea5b43" candidate_uuid: "549ef01dece445ddbd79862392dd53fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bda3fbdf6ef4f66aee8ae42d035bb7e" is_pre_election: true
I20240507 02:06:49.690789 1362 raft_consensus.cc:2454] T 8e698acfb3f848d2be86024d62ea5b43 P 8bda3fbdf6ef4f66aee8ae42d035bb7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 549ef01dece445ddbd79862392dd53fd in term 0.
I20240507 02:06:49.691193 1438 raft_consensus.cc:504] T 8e698acfb3f848d2be86024d62ea5b43 P 549ef01dece445ddbd79862392dd53fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "549ef01dece445ddbd79862392dd53fd" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43451 } } peers { permanent_uuid: "79f1aff9a44840d6959b5ac8e4d072e9" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 34747 } } peers { permanent_uuid: "8bda3fbdf6ef4f66aee8ae42d035bb7e" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 45181 } }
I20240507 02:06:49.691284 1438 leader_election.cc:288] T 8e698acfb3f848d2be86024d62ea5b43 P 549ef01dece445ddbd79862392dd53fd [CANDIDATE]: Term 1 election: Requested vote from peers 79f1aff9a44840d6959b5ac8e4d072e9 (127.31.131.66:34747), 8bda3fbdf6ef4f66aee8ae42d035bb7e (127.31.131.67:45181)
I20240507 02:06:49.691457 1220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e698acfb3f848d2be86024d62ea5b43" candidate_uuid: "549ef01dece445ddbd79862392dd53fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79f1aff9a44840d6959b5ac8e4d072e9"
I20240507 02:06:49.691490 1362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e698acfb3f848d2be86024d62ea5b43" candidate_uuid: "549ef01dece445ddbd79862392dd53fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bda3fbdf6ef4f66aee8ae42d035bb7e"
I20240507 02:06:49.691535 1220 raft_consensus.cc:3037] T 8e698acfb3f848d2be86024d62ea5b43 P 79f1aff9a44840d6959b5ac8e4d072e9 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:06:49.691545 1362 raft_consensus.cc:3037] T 8e698acfb3f848d2be86024d62ea5b43 P 8bda3fbdf6ef4f66aee8ae42d035bb7e [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:06:49.692210 1220 raft_consensus.cc:2454] T 8e698acfb3f848d2be86024d62ea5b43 P 79f1aff9a44840d6959b5ac8e4d072e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 549ef01dece445ddbd79862392dd53fd in term 1.
I20240507 02:06:49.692210 1362 raft_consensus.cc:2454] T 8e698acfb3f848d2be86024d62ea5b43 P 8bda3fbdf6ef4f66aee8ae42d035bb7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 549ef01dece445ddbd79862392dd53fd in term 1.
I20240507 02:06:49.692363 1012 leader_election.cc:302] T 8e698acfb3f848d2be86024d62ea5b43 P 549ef01dece445ddbd79862392dd53fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 549ef01dece445ddbd79862392dd53fd, 8bda3fbdf6ef4f66aee8ae42d035bb7e; no voters:
I20240507 02:06:49.692447 1438 raft_consensus.cc:2781] T 8e698acfb3f848d2be86024d62ea5b43 P 549ef01dece445ddbd79862392dd53fd [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:06:49.692497 1438 raft_consensus.cc:686] T 8e698acfb3f848d2be86024d62ea5b43 P 549ef01dece445ddbd79862392dd53fd [term 1 LEADER]: Becoming Leader. State: Replica: 549ef01dece445ddbd79862392dd53fd, State: Running, Role: LEADER
I20240507 02:06:49.692548 1438 consensus_queue.cc:237] T 8e698acfb3f848d2be86024d62ea5b43 P 549ef01dece445ddbd79862392dd53fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "549ef01dece445ddbd79862392dd53fd" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43451 } } peers { permanent_uuid: "79f1aff9a44840d6959b5ac8e4d072e9" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 34747 } } peers { permanent_uuid: "8bda3fbdf6ef4f66aee8ae42d035bb7e" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 45181 } }
I20240507 02:06:49.693023 918 catalog_manager.cc:5484] T 8e698acfb3f848d2be86024d62ea5b43 P 549ef01dece445ddbd79862392dd53fd reported cstate change: term changed from 0 to 1, leader changed from <none> to 549ef01dece445ddbd79862392dd53fd (127.31.131.65). New cstate: current_term: 1 leader_uuid: "549ef01dece445ddbd79862392dd53fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "549ef01dece445ddbd79862392dd53fd" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79f1aff9a44840d6959b5ac8e4d072e9" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 34747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8bda3fbdf6ef4f66aee8ae42d035bb7e" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 45181 } health_report { overall_health: UNKNOWN } } }
May 07 02:06:49 dist-test-slave-v0bg krb5kdc[874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047608, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.126@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:06:49.722309 972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:49.711227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55956 (local address 127.31.131.126:44459)
0507 02:06:49.711269 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:49.711271 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:49.711366 (+ 95us) server_negotiation.cc:407] Connection header received
0507 02:06:49.711439 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:49.711439 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:49.711448 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:49.711483 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:06:49.712264 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:49.712729 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:49.713485 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:49.713589 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:49.720205 (+ 6616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:06:49.720209 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:06:49.720210 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:06:49.720215 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:06:49.721436 (+ 1221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:49.721735 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:49.721736 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:49.721736 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:49.721755 (+ 19us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:49.721869 (+ 114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:49.721869 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:49.721870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:49.721948 (+ 78us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:06:49.721999 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:49.722116 (+ 117us) server_negotiation.cc:299] Negotiation successful
0507 02:06:49.722196 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20240507 02:06:49.722378 1463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:49.711140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:44459 (local address 127.0.0.1:55956)
0507 02:06:49.711287 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0507 02:06:49.711299 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0507 02:06:49.711376 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:06:49.711505 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:06:49.711506 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:06:49.711940 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:06:49.712193 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:49.712196 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:49.712800 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:06:49.712801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:06:49.713413 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:49.713416 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:49.713495 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:49.718538 (+ 5043us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:06:49.718543 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:06:49.720120 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:06:49.721503 (+ 1383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:06:49.721509 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:06:49.721512 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:06:49.721665 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:06:49.721801 (+ 136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:06:49.721802 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:06:49.721802 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:06:49.721826 (+ 24us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:06:49.721994 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:06:49.721999 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:06:49.722038 (+ 39us) client_negotiation.cc:769] Sending connection context
0507 02:06:49.722092 (+ 54us) client_negotiation.cc:240] Negotiation successful
0507 02:06:49.722261 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":73,"thread_start_us":78,"threads_started":1}
I20240507 02:06:49.724035 918 catalog_manager.cc:1904] Servicing CreateTable request from {username='kudu', principal='kudu/127.31.131.126@KRBTEST.COM'} at 127.0.0.1:55956:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20240507 02:06:49.724211 918 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 02:06:49.725548 1200 tablet_service.cc:1467] Processing CreateTablet for tablet 7152a064c7694ba5881ecebbc67759b3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=71984683a8b544e0812609d4b7c94a3e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20240507 02:06:49.725629 1342 tablet_service.cc:1467] Processing CreateTablet for tablet 7152a064c7694ba5881ecebbc67759b3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=71984683a8b544e0812609d4b7c94a3e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20240507 02:06:49.725642 1058 tablet_service.cc:1467] Processing CreateTablet for tablet 7152a064c7694ba5881ecebbc67759b3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=71984683a8b544e0812609d4b7c94a3e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20240507 02:06:49.725703 1200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7152a064c7694ba5881ecebbc67759b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:06:49.725728 1342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7152a064c7694ba5881ecebbc67759b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:06:49.725770 1058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7152a064c7694ba5881ecebbc67759b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:06:49.727051 1429 tablet_bootstrap.cc:492] T 7152a064c7694ba5881ecebbc67759b3 P 8bda3fbdf6ef4f66aee8ae42d035bb7e: Bootstrap starting.
I20240507 02:06:49.727051 1428 tablet_bootstrap.cc:492] T 7152a064c7694ba5881ecebbc67759b3 P 79f1aff9a44840d6959b5ac8e4d072e9: Bootstrap starting.
I20240507 02:06:49.727208 1431 tablet_bootstrap.cc:492] T 7152a064c7694ba5881ecebbc67759b3 P 549ef01dece445ddbd79862392dd53fd: Bootstrap starting.
I20240507 02:06:49.727407 1428 tablet_bootstrap.cc:654] T 7152a064c7694ba5881ecebbc67759b3 P 79f1aff9a44840d6959b5ac8e4d072e9: Neither blocks nor log segments found. Creating new log.
I20240507 02:06:49.727407 1429 tablet_bootstrap.cc:654] T 7152a064c7694ba5881ecebbc67759b3 P 8bda3fbdf6ef4f66aee8ae42d035bb7e: Neither blocks nor log segments found. Creating new log.
I20240507 02:06:49.727577 1431 tablet_bootstrap.cc:654] T 7152a064c7694ba5881ecebbc67759b3 P 549ef01dece445ddbd79862392dd53fd: Neither blocks nor log segments found. Creating new log.
I20240507 02:06:49.728156 1431 tablet_bootstrap.cc:492] T 7152a064c7694ba5881ecebbc67759b3 P 549ef01dece445ddbd79862392dd53fd: No bootstrap required, opened a new log
I20240507 02:06:49.728214 1431 ts_tablet_manager.cc:1367] T 7152a064c7694ba5881ecebbc67759b3 P 549ef01dece445ddbd79862392dd53fd: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240507 02:06:49.728336 1431 raft_consensus.cc:348] T 7152a064c7694ba5881ecebbc67759b3 P 549ef01dece445ddbd79862392dd53fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "549ef01dece445ddbd79862392dd53fd" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43451 } } peers { permanent_uuid: "79f1aff9a44840d6959b5ac8e4d072e9" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 34747 } } peers { permanent_uuid: "8bda3fbdf6ef4f66aee8ae42d035bb7e" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 45181 } }
I20240507 02:06:49.728390 1431 raft_consensus.cc:374] T 7152a064c7694ba5881ecebbc67759b3 P 549ef01dece445ddbd79862392dd53fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:06:49.728407 1431 raft_consensus.cc:729] T 7152a064c7694ba5881ecebbc67759b3 P 549ef01dece445ddbd79862392dd53fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 549ef01dece445ddbd79862392dd53fd, State: Initialized, Role: FOLLOWER
I20240507 02:06:49.728444 1431 consensus_queue.cc:260] T 7152a064c7694ba5881ecebbc67759b3 P 549ef01dece445ddbd79862392dd53fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "549ef01dece445ddbd79862392dd53fd" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43451 } } peers { permanent_uuid: "79f1aff9a44840d6959b5ac8e4d072e9" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 34747 } } peers { permanent_uuid: "8bda3fbdf6ef4f66aee8ae42d035bb7e" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 45181 } }
I20240507 02:06:49.728524 1431 ts_tablet_manager.cc:1398] T 7152a064c7694ba5881ecebbc67759b3 P 549ef01dece445ddbd79862392dd53fd: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:49.728619 1438 tablet_replica.cc:439] T 7152a064c7694ba5881ecebbc67759b3 P 549ef01dece445ddbd79862392dd53fd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "549ef01dece445ddbd79862392dd53fd" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43451 } } peers { permanent_uuid: "79f1aff9a44840d6959b5ac8e4d072e9" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 34747 } } peers { permanent_uuid: "8bda3fbdf6ef4f66aee8ae42d035bb7e" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 45181 } } }
I20240507 02:06:49.728636 1428 tablet_bootstrap.cc:492] T 7152a064c7694ba5881ecebbc67759b3 P 79f1aff9a44840d6959b5ac8e4d072e9: No bootstrap required, opened a new log
I20240507 02:06:49.728668 1438 tablet_replica.cc:442] T 7152a064c7694ba5881ecebbc67759b3 P 549ef01dece445ddbd79862392dd53fd: This TxnStatusTablet replica's current role is: FOLLOWER
I20240507 02:06:49.728678 1428 ts_tablet_manager.cc:1367] T 7152a064c7694ba5881ecebbc67759b3 P 79f1aff9a44840d6959b5ac8e4d072e9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240507 02:06:49.728814 1428 raft_consensus.cc:348] T 7152a064c7694ba5881ecebbc67759b3 P 79f1aff9a44840d6959b5ac8e4d072e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "549ef01dece445ddbd79862392dd53fd" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43451 } } peers { permanent_uuid: "79f1aff9a44840d6959b5ac8e4d072e9" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 34747 } } peers { permanent_uuid: "8bda3fbdf6ef4f66aee8ae42d035bb7e" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 45181 } }
I20240507 02:06:49.728880 1428 raft_consensus.cc:374] T 7152a064c7694ba5881ecebbc67759b3 P 79f1aff9a44840d6959b5ac8e4d072e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:06:49.728899 1428 raft_consensus.cc:729] T 7152a064c7694ba5881ecebbc67759b3 P 79f1aff9a44840d6959b5ac8e4d072e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79f1aff9a44840d6959b5ac8e4d072e9, State: Initialized, Role: FOLLOWER
I20240507 02:06:49.728950 1428 consensus_queue.cc:260] T 7152a064c7694ba5881ecebbc67759b3 P 79f1aff9a44840d6959b5ac8e4d072e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "549ef01dece445ddbd79862392dd53fd" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43451 } } peers { permanent_uuid: "79f1aff9a44840d6959b5ac8e4d072e9" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 34747 } } peers { permanent_uuid: "8bda3fbdf6ef4f66aee8ae42d035bb7e" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 45181 } }
I20240507 02:06:49.729034 1428 ts_tablet_manager.cc:1398] T 7152a064c7694ba5881ecebbc67759b3 P 79f1aff9a44840d6959b5ac8e4d072e9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:49.729156 1434 tablet_replica.cc:439] T 7152a064c7694ba5881ecebbc67759b3 P 79f1aff9a44840d6959b5ac8e4d072e9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "549ef01dece445ddbd79862392dd53fd" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43451 } } peers { permanent_uuid: "79f1aff9a44840d6959b5ac8e4d072e9" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 34747 } } peers { permanent_uuid: "8bda3fbdf6ef4f66aee8ae42d035bb7e" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 45181 } } }
I20240507 02:06:49.729234 1434 tablet_replica.cc:442] T 7152a064c7694ba5881ecebbc67759b3 P 79f1aff9a44840d6959b5ac8e4d072e9: This TxnStatusTablet replica's current role is: FOLLOWER
I20240507 02:06:49.729846 1429 tablet_bootstrap.cc:492] T 7152a064c7694ba5881ecebbc67759b3 P 8bda3fbdf6ef4f66aee8ae42d035bb7e: No bootstrap required, opened a new log
I20240507 02:06:49.729914 1429 ts_tablet_manager.cc:1367] T 7152a064c7694ba5881ecebbc67759b3 P 8bda3fbdf6ef4f66aee8ae42d035bb7e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240507 02:06:49.730073 1429 raft_consensus.cc:348] T 7152a064c7694ba5881ecebbc67759b3 P 8bda3fbdf6ef4f66aee8ae42d035bb7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "549ef01dece445ddbd79862392dd53fd" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43451 } } peers { permanent_uuid: "79f1aff9a44840d6959b5ac8e4d072e9" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 34747 } } peers { permanent_uuid: "8bda3fbdf6ef4f66aee8ae42d035bb7e" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 45181 } }
I20240507 02:06:49.730139 1429 raft_consensus.cc:374] T 7152a064c7694ba5881ecebbc67759b3 P 8bda3fbdf6ef4f66aee8ae42d035bb7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:06:49.730160 1429 raft_consensus.cc:729] T 7152a064c7694ba5881ecebbc67759b3 P 8bda3fbdf6ef4f66aee8ae42d035bb7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bda3fbdf6ef4f66aee8ae42d035bb7e, State: Initialized, Role: FOLLOWER
I20240507 02:06:49.730206 1429 consensus_queue.cc:260] T 7152a064c7694ba5881ecebbc67759b3 P 8bda3fbdf6ef4f66aee8ae42d035bb7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "549ef01dece445ddbd79862392dd53fd" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43451 } } peers { permanent_uuid: "79f1aff9a44840d6959b5ac8e4d072e9" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 34747 } } peers { permanent_uuid: "8bda3fbdf6ef4f66aee8ae42d035bb7e" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 45181 } }
I20240507 02:06:49.730296 1429 ts_tablet_manager.cc:1398] T 7152a064c7694ba5881ecebbc67759b3 P 8bda3fbdf6ef4f66aee8ae42d035bb7e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20240507 02:06:49.730418 1435 tablet_replica.cc:439] T 7152a064c7694ba5881ecebbc67759b3 P 8bda3fbdf6ef4f66aee8ae42d035bb7e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "549ef01dece445ddbd79862392dd53fd" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43451 } } peers { permanent_uuid: "79f1aff9a44840d6959b5ac8e4d072e9" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 34747 } } peers { permanent_uuid: "8bda3fbdf6ef4f66aee8ae42d035bb7e" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 45181 } } }
I20240507 02:06:49.730487 1435 tablet_replica.cc:442] T 7152a064c7694ba5881ecebbc67759b3 P 8bda3fbdf6ef4f66aee8ae42d035bb7e: This TxnStatusTablet replica's current role is: FOLLOWER
I20240507 02:06:49.777799 1438 raft_consensus.cc:482] T 7152a064c7694ba5881ecebbc67759b3 P 549ef01dece445ddbd79862392dd53fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 02:06:49.777902 1438 raft_consensus.cc:504] T 7152a064c7694ba5881ecebbc67759b3 P 549ef01dece445ddbd79862392dd53fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "549ef01dece445ddbd79862392dd53fd" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43451 } } peers { permanent_uuid: "79f1aff9a44840d6959b5ac8e4d072e9" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 34747 } } peers { permanent_uuid: "8bda3fbdf6ef4f66aee8ae42d035bb7e" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 45181 } }
I20240507 02:06:49.778048 1438 leader_election.cc:288] T 7152a064c7694ba5881ecebbc67759b3 P 549ef01dece445ddbd79862392dd53fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79f1aff9a44840d6959b5ac8e4d072e9 (127.31.131.66:34747), 8bda3fbdf6ef4f66aee8ae42d035bb7e (127.31.131.67:45181)
I20240507 02:06:49.778239 1220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152a064c7694ba5881ecebbc67759b3" candidate_uuid: "549ef01dece445ddbd79862392dd53fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79f1aff9a44840d6959b5ac8e4d072e9" is_pre_election: true
I20240507 02:06:49.778244 1362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152a064c7694ba5881ecebbc67759b3" candidate_uuid: "549ef01dece445ddbd79862392dd53fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bda3fbdf6ef4f66aee8ae42d035bb7e" is_pre_election: true
I20240507 02:06:49.778335 1362 raft_consensus.cc:2454] T 7152a064c7694ba5881ecebbc67759b3 P 8bda3fbdf6ef4f66aee8ae42d035bb7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 549ef01dece445ddbd79862392dd53fd in term 0.
I20240507 02:06:49.778335 1220 raft_consensus.cc:2454] T 7152a064c7694ba5881ecebbc67759b3 P 79f1aff9a44840d6959b5ac8e4d072e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 549ef01dece445ddbd79862392dd53fd in term 0.
I20240507 02:06:49.778479 1012 leader_election.cc:302] T 7152a064c7694ba5881ecebbc67759b3 P 549ef01dece445ddbd79862392dd53fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 549ef01dece445ddbd79862392dd53fd, 8bda3fbdf6ef4f66aee8ae42d035bb7e; no voters:
I20240507 02:06:49.778578 1438 raft_consensus.cc:2781] T 7152a064c7694ba5881ecebbc67759b3 P 549ef01dece445ddbd79862392dd53fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 02:06:49.778633 1438 raft_consensus.cc:482] T 7152a064c7694ba5881ecebbc67759b3 P 549ef01dece445ddbd79862392dd53fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 02:06:49.778661 1438 raft_consensus.cc:3037] T 7152a064c7694ba5881ecebbc67759b3 P 549ef01dece445ddbd79862392dd53fd [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:06:49.779235 1438 raft_consensus.cc:504] T 7152a064c7694ba5881ecebbc67759b3 P 549ef01dece445ddbd79862392dd53fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "549ef01dece445ddbd79862392dd53fd" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43451 } } peers { permanent_uuid: "79f1aff9a44840d6959b5ac8e4d072e9" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 34747 } } peers { permanent_uuid: "8bda3fbdf6ef4f66aee8ae42d035bb7e" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 45181 } }
I20240507 02:06:49.779341 1438 leader_election.cc:288] T 7152a064c7694ba5881ecebbc67759b3 P 549ef01dece445ddbd79862392dd53fd [CANDIDATE]: Term 1 election: Requested vote from peers 79f1aff9a44840d6959b5ac8e4d072e9 (127.31.131.66:34747), 8bda3fbdf6ef4f66aee8ae42d035bb7e (127.31.131.67:45181)
I20240507 02:06:49.779475 1220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152a064c7694ba5881ecebbc67759b3" candidate_uuid: "549ef01dece445ddbd79862392dd53fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79f1aff9a44840d6959b5ac8e4d072e9"
I20240507 02:06:49.779496 1362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7152a064c7694ba5881ecebbc67759b3" candidate_uuid: "549ef01dece445ddbd79862392dd53fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bda3fbdf6ef4f66aee8ae42d035bb7e"
I20240507 02:06:49.779536 1220 raft_consensus.cc:3037] T 7152a064c7694ba5881ecebbc67759b3 P 79f1aff9a44840d6959b5ac8e4d072e9 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:06:49.779549 1362 raft_consensus.cc:3037] T 7152a064c7694ba5881ecebbc67759b3 P 8bda3fbdf6ef4f66aee8ae42d035bb7e [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:06:49.780053 1362 raft_consensus.cc:2454] T 7152a064c7694ba5881ecebbc67759b3 P 8bda3fbdf6ef4f66aee8ae42d035bb7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 549ef01dece445ddbd79862392dd53fd in term 1.
I20240507 02:06:49.780099 1220 raft_consensus.cc:2454] T 7152a064c7694ba5881ecebbc67759b3 P 79f1aff9a44840d6959b5ac8e4d072e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 549ef01dece445ddbd79862392dd53fd in term 1.
I20240507 02:06:49.780221 1012 leader_election.cc:302] T 7152a064c7694ba5881ecebbc67759b3 P 549ef01dece445ddbd79862392dd53fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 549ef01dece445ddbd79862392dd53fd, 8bda3fbdf6ef4f66aee8ae42d035bb7e; no voters:
I20240507 02:06:49.780300 1438 raft_consensus.cc:2781] T 7152a064c7694ba5881ecebbc67759b3 P 549ef01dece445ddbd79862392dd53fd [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:06:49.780349 1438 raft_consensus.cc:686] T 7152a064c7694ba5881ecebbc67759b3 P 549ef01dece445ddbd79862392dd53fd [term 1 LEADER]: Becoming Leader. State: Replica: 549ef01dece445ddbd79862392dd53fd, State: Running, Role: LEADER
I20240507 02:06:49.780392 1438 consensus_queue.cc:237] T 7152a064c7694ba5881ecebbc67759b3 P 549ef01dece445ddbd79862392dd53fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "549ef01dece445ddbd79862392dd53fd" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43451 } } peers { permanent_uuid: "79f1aff9a44840d6959b5ac8e4d072e9" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 34747 } } peers { permanent_uuid: "8bda3fbdf6ef4f66aee8ae42d035bb7e" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 45181 } }
I20240507 02:06:49.780648 1447 tablet_replica.cc:439] T 7152a064c7694ba5881ecebbc67759b3 P 549ef01dece445ddbd79862392dd53fd: TxnStatusTablet state changed. Reason: New leader 549ef01dece445ddbd79862392dd53fd. Latest consensus state: current_term: 1 leader_uuid: "549ef01dece445ddbd79862392dd53fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "549ef01dece445ddbd79862392dd53fd" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43451 } } peers { permanent_uuid: "79f1aff9a44840d6959b5ac8e4d072e9" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 34747 } } peers { permanent_uuid: "8bda3fbdf6ef4f66aee8ae42d035bb7e" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 45181 } } }
I20240507 02:06:49.780732 1447 tablet_replica.cc:442] T 7152a064c7694ba5881ecebbc67759b3 P 549ef01dece445ddbd79862392dd53fd: This TxnStatusTablet replica's current role is: LEADER
I20240507 02:06:49.780934 918 catalog_manager.cc:5484] T 7152a064c7694ba5881ecebbc67759b3 P 549ef01dece445ddbd79862392dd53fd reported cstate change: term changed from 0 to 1, leader changed from <none> to 549ef01dece445ddbd79862392dd53fd (127.31.131.65). New cstate: current_term: 1 leader_uuid: "549ef01dece445ddbd79862392dd53fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "549ef01dece445ddbd79862392dd53fd" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79f1aff9a44840d6959b5ac8e4d072e9" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 34747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8bda3fbdf6ef4f66aee8ae42d035bb7e" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 45181 } health_report { overall_health: UNKNOWN } } }
I20240507 02:06:49.781188 1468 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20240507 02:06:49.781275 1468 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20240507 02:06:49.788148 1463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:49.786218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.65:43451 (local address 127.0.0.1:36964)
0507 02:06:49.786335 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0507 02:06:49.786345 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0507 02:06:49.786399 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:06:49.786534 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:06:49.786534 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:06:49.786536 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0507 02:06:49.786713 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:49.786715 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:49.787274 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:06:49.787275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:06:49.787687 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:49.787688 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:49.787748 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:49.787766 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0507 02:06:49.787989 (+ 223us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0507 02:06:49.787998 (+ 9us) client_negotiation.cc:769] Sending connection context
0507 02:06:49.788038 (+ 40us) client_negotiation.cc:240] Negotiation successful
0507 02:06:49.788083 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20240507 02:06:49.788143 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:49.786269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36964 (local address 127.31.131.65:43451)
0507 02:06:49.786318 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:49.786320 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:49.786391 (+ 71us) server_negotiation.cc:407] Connection header received
0507 02:06:49.786473 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:49.786474 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:49.786482 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:49.786520 (+ 38us) server_negotiation.cc:226] Negotiated authn=TOKEN
0507 02:06:49.786774 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:49.787213 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:49.787759 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:49.787897 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:49.787913 (+ 16us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0507 02:06:49.787936 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0507 02:06:49.787977 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:49.788051 (+ 74us) server_negotiation.cc:299] Negotiation successful
0507 02:06:49.788071 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20240507 02:06:49.789053 1362 raft_consensus.cc:1261] T 7152a064c7694ba5881ecebbc67759b3 P 8bda3fbdf6ef4f66aee8ae42d035bb7e [term 1 FOLLOWER]: Refusing update from remote peer 549ef01dece445ddbd79862392dd53fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 02:06:49.789052 1220 raft_consensus.cc:1261] T 7152a064c7694ba5881ecebbc67759b3 P 79f1aff9a44840d6959b5ac8e4d072e9 [term 1 FOLLOWER]: Refusing update from remote peer 549ef01dece445ddbd79862392dd53fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 02:06:49.789254 1447 consensus_queue.cc:1035] T 7152a064c7694ba5881ecebbc67759b3 P 549ef01dece445ddbd79862392dd53fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8bda3fbdf6ef4f66aee8ae42d035bb7e" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 45181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 02:06:49.789364 1438 consensus_queue.cc:1035] T 7152a064c7694ba5881ecebbc67759b3 P 549ef01dece445ddbd79862392dd53fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "79f1aff9a44840d6959b5ac8e4d072e9" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 02:06:49.789726 1434 tablet_replica.cc:439] T 7152a064c7694ba5881ecebbc67759b3 P 79f1aff9a44840d6959b5ac8e4d072e9: TxnStatusTablet state changed. Reason: New leader 549ef01dece445ddbd79862392dd53fd. Latest consensus state: current_term: 1 leader_uuid: "549ef01dece445ddbd79862392dd53fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "549ef01dece445ddbd79862392dd53fd" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43451 } } peers { permanent_uuid: "79f1aff9a44840d6959b5ac8e4d072e9" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 34747 } } peers { permanent_uuid: "8bda3fbdf6ef4f66aee8ae42d035bb7e" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 45181 } } }
I20240507 02:06:49.789808 1434 tablet_replica.cc:442] T 7152a064c7694ba5881ecebbc67759b3 P 79f1aff9a44840d6959b5ac8e4d072e9: This TxnStatusTablet replica's current role is: FOLLOWER
I20240507 02:06:49.789806 1435 tablet_replica.cc:439] T 7152a064c7694ba5881ecebbc67759b3 P 8bda3fbdf6ef4f66aee8ae42d035bb7e: TxnStatusTablet state changed. Reason: New leader 549ef01dece445ddbd79862392dd53fd. Latest consensus state: current_term: 1 leader_uuid: "549ef01dece445ddbd79862392dd53fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "549ef01dece445ddbd79862392dd53fd" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43451 } } peers { permanent_uuid: "79f1aff9a44840d6959b5ac8e4d072e9" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 34747 } } peers { permanent_uuid: "8bda3fbdf6ef4f66aee8ae42d035bb7e" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 45181 } } }
I20240507 02:06:49.789881 1435 tablet_replica.cc:442] T 7152a064c7694ba5881ecebbc67759b3 P 8bda3fbdf6ef4f66aee8ae42d035bb7e: This TxnStatusTablet replica's current role is: FOLLOWER
I20240507 02:06:49.790320 1438 tablet_replica.cc:439] T 7152a064c7694ba5881ecebbc67759b3 P 549ef01dece445ddbd79862392dd53fd: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "549ef01dece445ddbd79862392dd53fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "549ef01dece445ddbd79862392dd53fd" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43451 } } peers { permanent_uuid: "79f1aff9a44840d6959b5ac8e4d072e9" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 34747 } } peers { permanent_uuid: "8bda3fbdf6ef4f66aee8ae42d035bb7e" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 45181 } } }
I20240507 02:06:49.790387 1438 tablet_replica.cc:442] T 7152a064c7694ba5881ecebbc67759b3 P 549ef01dece445ddbd79862392dd53fd: This TxnStatusTablet replica's current role is: LEADER
I20240507 02:06:49.790462 1441 mvcc.cc:205] Tried to move back new op lower bound from 7024835009694674944 to 7024835009660903424. Current Snapshot: MvccSnapshot[applied={T|T < 7024835009694674944}]
I20240507 02:06:49.790967 1436 tablet_replica.cc:439] T 7152a064c7694ba5881ecebbc67759b3 P 549ef01dece445ddbd79862392dd53fd: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "549ef01dece445ddbd79862392dd53fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "549ef01dece445ddbd79862392dd53fd" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43451 } } peers { permanent_uuid: "79f1aff9a44840d6959b5ac8e4d072e9" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 34747 } } peers { permanent_uuid: "8bda3fbdf6ef4f66aee8ae42d035bb7e" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 45181 } } }
I20240507 02:06:49.791033 1436 tablet_replica.cc:442] T 7152a064c7694ba5881ecebbc67759b3 P 549ef01dece445ddbd79862392dd53fd: This TxnStatusTablet replica's current role is: LEADER
I20240507 02:06:49.790992 1434 tablet_replica.cc:439] T 7152a064c7694ba5881ecebbc67759b3 P 79f1aff9a44840d6959b5ac8e4d072e9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "549ef01dece445ddbd79862392dd53fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "549ef01dece445ddbd79862392dd53fd" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43451 } } peers { permanent_uuid: "79f1aff9a44840d6959b5ac8e4d072e9" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 34747 } } peers { permanent_uuid: "8bda3fbdf6ef4f66aee8ae42d035bb7e" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 45181 } } }
I20240507 02:06:49.791056 1434 tablet_replica.cc:442] T 7152a064c7694ba5881ecebbc67759b3 P 79f1aff9a44840d6959b5ac8e4d072e9: This TxnStatusTablet replica's current role is: FOLLOWER
I20240507 02:06:49.791589 1435 tablet_replica.cc:439] T 7152a064c7694ba5881ecebbc67759b3 P 8bda3fbdf6ef4f66aee8ae42d035bb7e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "549ef01dece445ddbd79862392dd53fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "549ef01dece445ddbd79862392dd53fd" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43451 } } peers { permanent_uuid: "79f1aff9a44840d6959b5ac8e4d072e9" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 34747 } } peers { permanent_uuid: "8bda3fbdf6ef4f66aee8ae42d035bb7e" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 45181 } } }
I20240507 02:06:49.791671 1435 tablet_replica.cc:442] T 7152a064c7694ba5881ecebbc67759b3 P 8bda3fbdf6ef4f66aee8ae42d035bb7e: This TxnStatusTablet replica's current role is: FOLLOWER
I20240507 02:06:49.796609 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:49.794504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:45869 (local address 127.31.131.65:43451)
0507 02:06:49.794540 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:49.794541 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:49.794708 (+ 167us) server_negotiation.cc:407] Connection header received
0507 02:06:49.794764 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:49.794764 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:49.794771 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:49.794813 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0507 02:06:49.795096 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:49.795505 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:49.796295 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:49.796401 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:49.796415 (+ 14us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0507 02:06:49.796436 (+ 21us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0507 02:06:49.796475 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:49.796536 (+ 61us) server_negotiation.cc:299] Negotiation successful
0507 02:06:49.796551 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20240507 02:06:49.796697 1474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:49.794462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.65:43451 (local address 127.31.131.65:45869)
0507 02:06:49.794637 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0507 02:06:49.794648 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0507 02:06:49.794710 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:06:49.794827 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:06:49.794828 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:06:49.794830 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0507 02:06:49.795045 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:49.795047 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:49.795560 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:06:49.795561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:06:49.796234 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:49.796236 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:49.796299 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:49.796315 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0507 02:06:49.796491 (+ 176us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0507 02:06:49.796492 (+ 1us) client_negotiation.cc:769] Sending connection context
0507 02:06:49.796532 (+ 40us) client_negotiation.cc:240] Negotiation successful
0507 02:06:49.796575 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":73,"threads_started":1}
I20240507 02:06:49.798504 1220 raft_consensus.cc:1261] T 8e698acfb3f848d2be86024d62ea5b43 P 79f1aff9a44840d6959b5ac8e4d072e9 [term 1 FOLLOWER]: Refusing update from remote peer 549ef01dece445ddbd79862392dd53fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 02:06:49.798506 1362 raft_consensus.cc:1261] T 8e698acfb3f848d2be86024d62ea5b43 P 8bda3fbdf6ef4f66aee8ae42d035bb7e [term 1 FOLLOWER]: Refusing update from remote peer 549ef01dece445ddbd79862392dd53fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 02:06:49.798732 1436 consensus_queue.cc:1035] T 8e698acfb3f848d2be86024d62ea5b43 P 549ef01dece445ddbd79862392dd53fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8bda3fbdf6ef4f66aee8ae42d035bb7e" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 45181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 02:06:49.798817 1438 consensus_queue.cc:1035] T 8e698acfb3f848d2be86024d62ea5b43 P 549ef01dece445ddbd79862392dd53fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "79f1aff9a44840d6959b5ac8e4d072e9" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 02:06:49.811193 919 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55950:
table { table_name: "test-table" } modify_external_catalogs: true
I20240507 02:06:49.811275 919 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55950:
table { table_name: "test-table" } modify_external_catalogs: true
I20240507 02:06:49.811798 919 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c10b07de71ff43efb4d7a1d211569a29: Sending DeleteTablet for 3 replicas of tablet 8e698acfb3f848d2be86024d62ea5b43
I20240507 02:06:49.812052 1057 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e698acfb3f848d2be86024d62ea5b43 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:06:49 UTC) from {username='kudu', principal='kudu/127.31.131.126@KRBTEST.COM'} at 127.0.0.1:36958
I20240507 02:06:49.812050 1200 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e698acfb3f848d2be86024d62ea5b43 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:06:49 UTC) from {username='kudu', principal='kudu/127.31.131.126@KRBTEST.COM'} at 127.0.0.1:57786
I20240507 02:06:49.812064 1342 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e698acfb3f848d2be86024d62ea5b43 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:06:49 UTC) from {username='kudu', principal='kudu/127.31.131.126@KRBTEST.COM'} at 127.0.0.1:42822
I20240507 02:06:49.812131 1452 tablet_replica.cc:330] T 8e698acfb3f848d2be86024d62ea5b43 P 549ef01dece445ddbd79862392dd53fd: stopping tablet replica
I20240507 02:06:49.812142 1451 tablet_replica.cc:330] T 8e698acfb3f848d2be86024d62ea5b43 P 8bda3fbdf6ef4f66aee8ae42d035bb7e: stopping tablet replica
I20240507 02:06:49.812147 1453 tablet_replica.cc:330] T 8e698acfb3f848d2be86024d62ea5b43 P 79f1aff9a44840d6959b5ac8e4d072e9: stopping tablet replica
I20240507 02:06:49.812168 1452 raft_consensus.cc:2229] T 8e698acfb3f848d2be86024d62ea5b43 P 549ef01dece445ddbd79862392dd53fd [term 1 LEADER]: Raft consensus shutting down.
I20240507 02:06:49.812188 1451 raft_consensus.cc:2229] T 8e698acfb3f848d2be86024d62ea5b43 P 8bda3fbdf6ef4f66aee8ae42d035bb7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 02:06:49.812191 1453 raft_consensus.cc:2229] T 8e698acfb3f848d2be86024d62ea5b43 P 79f1aff9a44840d6959b5ac8e4d072e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 02:06:49.812204 1452 raft_consensus.cc:2258] T 8e698acfb3f848d2be86024d62ea5b43 P 549ef01dece445ddbd79862392dd53fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 02:06:49.812218 1453 raft_consensus.cc:2258] T 8e698acfb3f848d2be86024d62ea5b43 P 79f1aff9a44840d6959b5ac8e4d072e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 02:06:49.812218 1451 raft_consensus.cc:2258] T 8e698acfb3f848d2be86024d62ea5b43 P 8bda3fbdf6ef4f66aee8ae42d035bb7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 02:06:49.812515 1453 ts_tablet_manager.cc:1875] T 8e698acfb3f848d2be86024d62ea5b43 P 79f1aff9a44840d6959b5ac8e4d072e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 02:06:49.812580 1452 ts_tablet_manager.cc:1875] T 8e698acfb3f848d2be86024d62ea5b43 P 549ef01dece445ddbd79862392dd53fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 02:06:49.813016 1451 ts_tablet_manager.cc:1875] T 8e698acfb3f848d2be86024d62ea5b43 P 8bda3fbdf6ef4f66aee8ae42d035bb7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 02:06:49.814127 1453 ts_tablet_manager.cc:1888] T 8e698acfb3f848d2be86024d62ea5b43 P 79f1aff9a44840d6959b5ac8e4d072e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240507 02:06:49.814121 1452 ts_tablet_manager.cc:1888] T 8e698acfb3f848d2be86024d62ea5b43 P 549ef01dece445ddbd79862392dd53fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240507 02:06:49.814174 1452 log.cc:1196] T 8e698acfb3f848d2be86024d62ea5b43 P 549ef01dece445ddbd79862392dd53fd: Deleting WAL directory at /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/minicluster-data/ts-0/wal/wals/8e698acfb3f848d2be86024d62ea5b43
I20240507 02:06:49.814174 1453 log.cc:1196] T 8e698acfb3f848d2be86024d62ea5b43 P 79f1aff9a44840d6959b5ac8e4d072e9: Deleting WAL directory at /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/minicluster-data/ts-1/wal/wals/8e698acfb3f848d2be86024d62ea5b43
I20240507 02:06:49.814368 1452 ts_tablet_manager.cc:1909] T 8e698acfb3f848d2be86024d62ea5b43 P 549ef01dece445ddbd79862392dd53fd: Deleting consensus metadata
I20240507 02:06:49.814393 1453 ts_tablet_manager.cc:1909] T 8e698acfb3f848d2be86024d62ea5b43 P 79f1aff9a44840d6959b5ac8e4d072e9: Deleting consensus metadata
I20240507 02:06:49.814445 1451 ts_tablet_manager.cc:1888] T 8e698acfb3f848d2be86024d62ea5b43 P 8bda3fbdf6ef4f66aee8ae42d035bb7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240507 02:06:49.814513 1451 log.cc:1196] T 8e698acfb3f848d2be86024d62ea5b43 P 8bda3fbdf6ef4f66aee8ae42d035bb7e: Deleting WAL directory at /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1715047604443657-32269-0/minicluster-data/ts-2/wal/wals/8e698acfb3f848d2be86024d62ea5b43
I20240507 02:06:49.814718 1451 ts_tablet_manager.cc:1909] T 8e698acfb3f848d2be86024d62ea5b43 P 8bda3fbdf6ef4f66aee8ae42d035bb7e: Deleting consensus metadata
I20240507 02:06:49.814738 906 catalog_manager.cc:4830] TS 79f1aff9a44840d6959b5ac8e4d072e9 (127.31.131.66:34747): tablet 8e698acfb3f848d2be86024d62ea5b43 (table test-table [id=35eac029cf304da982b723194ace138b]) successfully deleted
I20240507 02:06:49.814797 906 catalog_manager.cc:4830] TS 549ef01dece445ddbd79862392dd53fd (127.31.131.65:43451): tablet 8e698acfb3f848d2be86024d62ea5b43 (table test-table [id=35eac029cf304da982b723194ace138b]) successfully deleted
I20240507 02:06:49.815093 906 catalog_manager.cc:4830] TS 8bda3fbdf6ef4f66aee8ae42d035bb7e (127.31.131.67:45181): tablet 8e698acfb3f848d2be86024d62ea5b43 (table test-table [id=35eac029cf304da982b723194ace138b]) successfully deleted
May 07 02:06:49 dist-test-slave-v0bg krb5kdc[874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047609, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.131.65@KRBTEST.COM
I20240507 02:06:49.821892 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:49.814847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36966 (local address 127.31.131.65:43451)
0507 02:06:49.814917 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:49.814919 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:49.815143 (+ 224us) server_negotiation.cc:407] Connection header received
0507 02:06:49.815240 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:49.815240 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:49.815252 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:49.815292 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:06:49.816073 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:49.816505 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:49.817247 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:49.817350 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:49.819591 (+ 2241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:06:49.819600 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:06:49.819600 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:06:49.819611 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:06:49.821032 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:49.821339 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:49.821341 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:49.821341 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:49.821367 (+ 26us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:49.821507 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:49.821507 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:49.821508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:49.821587 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:06:49.821687 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:49.821767 (+ 80us) server_negotiation.cc:299] Negotiation successful
0507 02:06:49.821835 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20240507 02:06:49.822180 1119 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36966
I20240507 02:06:49.827809 972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:49.822979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55972 (local address 127.31.131.126:44459)
0507 02:06:49.823021 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:49.823023 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:49.823055 (+ 32us) server_negotiation.cc:407] Connection header received
0507 02:06:49.823101 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:49.823102 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:49.823111 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:49.823163 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:06:49.823632 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:49.824381 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:49.824842 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:49.825007 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:49.825714 (+ 707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:06:49.825717 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:06:49.825717 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:06:49.825723 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:06:49.827063 (+ 1340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:49.827306 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:49.827307 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:49.827307 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:49.827328 (+ 21us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:49.827425 (+ 97us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:49.827426 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:49.827426 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:49.827516 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:06:49.827603 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:49.827658 (+ 55us) server_negotiation.cc:299] Negotiation successful
0507 02:06:49.827725 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20240507 02:06:49.827958 919 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55972
I20240507 02:06:49.828434 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 993
I20240507 02:06:49.834789 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 1134
I20240507 02:06:49.839015 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 1276
I20240507 02:06:49.842629 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 887
2024-05-07T02:06:49Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (1996 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 07 02:06:49 dist-test-slave-v0bg krb5kdc[1484](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 07 02:06:49 dist-test-slave-v0bg krb5kdc[1484](info): set up 2 sockets
May 07 02:06:49 dist-test-slave-v0bg krb5kdc[1484](info): commencing operation
krb5kdc: starting...
W20240507 02:06:50.727324 32269 mini_kdc.cc:121] Time spent starting KDC: real 0.866s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/krb5kdc/test-user.keytab.
May 07 02:06:50 dist-test-slave-v0bg krb5kdc[1484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047610, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-07T02:06:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-07T02:06:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.131.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab.
Entry for principal kudu/127.31.131.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab.
I20240507 02:06:50.826272 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:34169
--webserver_interface=127.31.131.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab
--principal=kudu/127.31.131.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:41621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.131.126:34169
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:06:50.900522 1497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:06:50.900701 1497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:06:50.900722 1497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:06:50.902012 1497 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240507 02:06:50.902050 1497 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20240507 02:06:50.902061 1497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:06:50.902073 1497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:06:50.902082 1497 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240507 02:06:50.902093 1497 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240507 02:06:50.903611 1497 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:41621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.131.126:34169
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab
--principal=kudu/127.31.131.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:34169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.131.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1497
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:06:50.903779 1497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:06:50.903908 1497 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:06:50.907352 1503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:06:50.907478 1497 server_base.cc:1034] running on GCE node
W20240507 02:06:50.907536 1502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:06:50.907393 1505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:06:50.907814 1497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:06:50.908021 1497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:06:50.909142 1497 hybrid_clock.cc:648] HybridClock initialized: now 1715047610909125 us; error 31 us; skew 500 ppm
May 07 02:06:50 dist-test-slave-v0bg krb5kdc[1484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047610, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:06:50.911291 1497 init.cc:377] Logged in from keytab as kudu/127.31.131.126@KRBTEST.COM (short username kudu)
I20240507 02:06:50.911660 1497 webserver.cc:460] Webserver started at http://127.31.131.126:46819/ using document root <none> and password file <none>
I20240507 02:06:50.911818 1497 fs_manager.cc:359] Metadata directory not provided
I20240507 02:06:50.911855 1497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:06:50.911952 1497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:06:50.912830 1497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/master-0/data/instance:
uuid: "27251517e7da4be8bebf6abfb08c2a57"
format_stamp: "Formatted at 2024-05-07 02:06:50 on dist-test-slave-v0bg"
server_key: "20679ef75d7558f842c2c4efe5d434c7"
server_key_iv: "86568460a1f303ef92a894735ffd534e"
server_key_version: "encryptionkey@0"
I20240507 02:06:50.913089 1497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/master-0/wal/instance:
uuid: "27251517e7da4be8bebf6abfb08c2a57"
format_stamp: "Formatted at 2024-05-07 02:06:50 on dist-test-slave-v0bg"
server_key: "20679ef75d7558f842c2c4efe5d434c7"
server_key_iv: "86568460a1f303ef92a894735ffd534e"
server_key_version: "encryptionkey@0"
I20240507 02:06:50.914105 1497 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240507 02:06:50.914697 1512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:50.914839 1497 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240507 02:06:50.914906 1497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/master-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/master-0/wal
uuid: "27251517e7da4be8bebf6abfb08c2a57"
format_stamp: "Formatted at 2024-05-07 02:06:50 on dist-test-slave-v0bg"
server_key: "20679ef75d7558f842c2c4efe5d434c7"
server_key_iv: "86568460a1f303ef92a894735ffd534e"
server_key_version: "encryptionkey@0"
I20240507 02:06:50.914960 1497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:06:50.922787 1497 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:06:50.924255 1497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:06:50.924356 1497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:06:50.928125 1497 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.126:34169
I20240507 02:06:50.928172 1574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.126:34169 every 8 connection(s)
I20240507 02:06:50.928428 1497 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
I20240507 02:06:50.928926 1575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:06:50.929314 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 1497
I20240507 02:06:50.929386 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/master-0/wal/instance
I20240507 02:06:50.929527 32269 external_mini_cluster.cc:1428] Setting key 0a4db4dd775f72d268e8eec5cffe1eed
I20240507 02:06:50.931095 1575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27251517e7da4be8bebf6abfb08c2a57: Bootstrap starting.
I20240507 02:06:50.931584 1575 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27251517e7da4be8bebf6abfb08c2a57: Neither blocks nor log segments found. Creating new log.
I20240507 02:06:50.931809 1575 log.cc:824] T 00000000000000000000000000000000 P 27251517e7da4be8bebf6abfb08c2a57: Log is configured to *not* fsync() on all Append() calls
I20240507 02:06:50.932391 1575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27251517e7da4be8bebf6abfb08c2a57: No bootstrap required, opened a new log
I20240507 02:06:50.934111 1575 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27251517e7da4be8bebf6abfb08c2a57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27251517e7da4be8bebf6abfb08c2a57" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 34169 } }
I20240507 02:06:50.934254 1575 raft_consensus.cc:374] T 00000000000000000000000000000000 P 27251517e7da4be8bebf6abfb08c2a57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:06:50.934280 1575 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27251517e7da4be8bebf6abfb08c2a57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27251517e7da4be8bebf6abfb08c2a57, State: Initialized, Role: FOLLOWER
I20240507 02:06:50.934382 1575 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27251517e7da4be8bebf6abfb08c2a57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27251517e7da4be8bebf6abfb08c2a57" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 34169 } }
I20240507 02:06:50.934449 1575 raft_consensus.cc:388] T 00000000000000000000000000000000 P 27251517e7da4be8bebf6abfb08c2a57 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 02:06:50.934481 1575 raft_consensus.cc:482] T 00000000000000000000000000000000 P 27251517e7da4be8bebf6abfb08c2a57 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
May 07 02:06:50 dist-test-slave-v0bg krb5kdc[1484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047610, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:06:50.934521 1575 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27251517e7da4be8bebf6abfb08c2a57 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:06:50.935025 1575 raft_consensus.cc:504] T 00000000000000000000000000000000 P 27251517e7da4be8bebf6abfb08c2a57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27251517e7da4be8bebf6abfb08c2a57" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 34169 } }
I20240507 02:06:50.935117 1575 leader_election.cc:302] T 00000000000000000000000000000000 P 27251517e7da4be8bebf6abfb08c2a57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 27251517e7da4be8bebf6abfb08c2a57; no voters:
I20240507 02:06:50.935258 1575 leader_election.cc:288] T 00000000000000000000000000000000 P 27251517e7da4be8bebf6abfb08c2a57 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240507 02:06:50.935289 1580 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 27251517e7da4be8bebf6abfb08c2a57 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:06:50.935454 1580 raft_consensus.cc:686] T 00000000000000000000000000000000 P 27251517e7da4be8bebf6abfb08c2a57 [term 1 LEADER]: Becoming Leader. State: Replica: 27251517e7da4be8bebf6abfb08c2a57, State: Running, Role: LEADER
I20240507 02:06:50.935463 1575 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27251517e7da4be8bebf6abfb08c2a57 [sys.catalog]: configured and running, proceeding with master startup.
I20240507 02:06:50.935570 1580 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27251517e7da4be8bebf6abfb08c2a57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27251517e7da4be8bebf6abfb08c2a57" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 34169 } }
I20240507 02:06:50.935858 1580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27251517e7da4be8bebf6abfb08c2a57 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27251517e7da4be8bebf6abfb08c2a57. Latest consensus state: current_term: 1 leader_uuid: "27251517e7da4be8bebf6abfb08c2a57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27251517e7da4be8bebf6abfb08c2a57" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 34169 } } }
I20240507 02:06:50.935871 1581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27251517e7da4be8bebf6abfb08c2a57 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "27251517e7da4be8bebf6abfb08c2a57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27251517e7da4be8bebf6abfb08c2a57" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 34169 } } }
I20240507 02:06:50.935927 1580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27251517e7da4be8bebf6abfb08c2a57 [sys.catalog]: This master's current role is: LEADER
I20240507 02:06:50.935940 1581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27251517e7da4be8bebf6abfb08c2a57 [sys.catalog]: This master's current role is: LEADER
I20240507 02:06:50.936095 1585 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 02:06:50.936735 1585 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240507 02:06:50.938213 1585 catalog_manager.cc:1347] Generated new cluster ID: 8d65a80237b04424abd0338373d5f7f9
I20240507 02:06:50.938277 1585 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240507 02:06:50.938336 1578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:50.929959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59100 (local address 127.31.131.126:34169)
0507 02:06:50.930105 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:50.930117 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:50.930132 (+ 15us) server_negotiation.cc:407] Connection header received
0507 02:06:50.930489 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:50.930494 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:50.930531 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:50.930669 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:06:50.931394 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:50.932118 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:50.932678 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:50.932882 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:50.934956 (+ 2074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:06:50.934964 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:06:50.934964 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:06:50.934979 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:06:50.936964 (+ 1985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:50.937304 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:50.937306 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:50.937307 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:50.937351 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:50.937494 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:50.937495 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:50.937495 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:50.937720 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:06:50.937864 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:50.937991 (+ 127us) server_negotiation.cc:299] Negotiation successful
0507 02:06:50.938114 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":73,"threads_started":1}
I20240507 02:06:50.946565 1585 catalog_manager.cc:1370] Generated new certificate authority record
I20240507 02:06:50.947146 1585 catalog_manager.cc:1504] Loading token signing keys...
I20240507 02:06:50.952304 1585 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 27251517e7da4be8bebf6abfb08c2a57: Generated new TSK 0
I20240507 02:06:50.952459 1585 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.131.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab.
Entry for principal kudu/127.31.131.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab.
I20240507 02:06:50.980862 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.65:0
--local_ip_for_outbound_sockets=127.31.131.65
--webserver_interface=127.31.131.65
--webserver_port=0
--tserver_master_addrs=127.31.131.126:34169
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab
--principal=kudu/127.31.131.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:41621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:06:51.056421 1601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:06:51.056612 1601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:06:51.056648 1601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:06:51.057915 1601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:06:51.057960 1601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:06:51.057989 1601 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20240507 02:06:51.058007 1601 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20240507 02:06:51.058041 1601 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.65
I20240507 02:06:51.059854 1601 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:41621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab
--principal=kudu/127.31.131.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.131.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.131.126:34169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1601
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.65
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:06:51.060046 1601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:06:51.060201 1601 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:06:51.064286 1610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:06:51.064407 1601 server_base.cc:1034] running on GCE node
W20240507 02:06:51.064282 1607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:06:51.064467 1608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:06:51.064640 1601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:06:51.064839 1601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:06:51.065963 1601 hybrid_clock.cc:648] HybridClock initialized: now 1715047611065952 us; error 30 us; skew 500 ppm
May 07 02:06:51 dist-test-slave-v0bg krb5kdc[1484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047611, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:06:51.068295 1601 init.cc:377] Logged in from keytab as kudu/127.31.131.65@KRBTEST.COM (short username kudu)
I20240507 02:06:51.068684 1601 webserver.cc:460] Webserver started at http://127.31.131.65:38475/ using document root <none> and password file <none>
I20240507 02:06:51.068850 1601 fs_manager.cc:359] Metadata directory not provided
I20240507 02:06:51.068887 1601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:06:51.069000 1601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:06:51.069911 1601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/ts-0/data/instance:
uuid: "eb01f5a060104233b45cb74c075401da"
format_stamp: "Formatted at 2024-05-07 02:06:51 on dist-test-slave-v0bg"
server_key: "3ada9697da2794cdd08e09b0c8c11134"
server_key_iv: "61fedb366ace3185327ad2f4ede4dabd"
server_key_version: "encryptionkey@0"
I20240507 02:06:51.070186 1601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/ts-0/wal/instance:
uuid: "eb01f5a060104233b45cb74c075401da"
format_stamp: "Formatted at 2024-05-07 02:06:51 on dist-test-slave-v0bg"
server_key: "3ada9697da2794cdd08e09b0c8c11134"
server_key_iv: "61fedb366ace3185327ad2f4ede4dabd"
server_key_version: "encryptionkey@0"
I20240507 02:06:51.071278 1601 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240507 02:06:51.071887 1617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:51.072062 1601 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240507 02:06:51.072151 1601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/ts-0/wal
uuid: "eb01f5a060104233b45cb74c075401da"
format_stamp: "Formatted at 2024-05-07 02:06:51 on dist-test-slave-v0bg"
server_key: "3ada9697da2794cdd08e09b0c8c11134"
server_key_iv: "61fedb366ace3185327ad2f4ede4dabd"
server_key_version: "encryptionkey@0"
I20240507 02:06:51.072232 1601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:06:51.087667 1601 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:06:51.089066 1601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:06:51.089155 1601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:06:51.089567 1601 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:06:51.089605 1601 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:51.089632 1601 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:06:51.089646 1601 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:51.094964 1601 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.65:33363
I20240507 02:06:51.095031 1732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.65:33363 every 8 connection(s)
I20240507 02:06:51.095268 1601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
May 07 02:06:51 dist-test-slave-v0bg krb5kdc[1484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047611, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.65@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:06:51.097946 1578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:51.089930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:47957 (local address 127.31.131.126:34169)
0507 02:06:51.089991 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:51.089994 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:51.090209 (+ 215us) server_negotiation.cc:407] Connection header received
0507 02:06:51.090815 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:51.090818 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:51.090830 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:51.090910 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:06:51.091838 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:51.092570 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:51.093311 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:51.093486 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:51.096015 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:06:51.096021 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:06:51.096021 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:06:51.096027 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:06:51.097092 (+ 1065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:51.097387 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:51.097387 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:51.097388 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:51.097403 (+ 15us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:51.097522 (+ 119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:51.097522 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:51.097522 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:51.097595 (+ 73us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:06:51.097637 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:51.097819 (+ 182us) server_negotiation.cc:299] Negotiation successful
0507 02:06:51.097882 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20240507 02:06:51.098028 1626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:51.089908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:34169 (local address 127.31.131.65:47957)
0507 02:06:51.090125 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0507 02:06:51.090144 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0507 02:06:51.090693 (+ 549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:06:51.090970 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:06:51.090974 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:06:51.091338 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:06:51.091741 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:51.091747 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:51.092665 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:06:51.092666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:06:51.093219 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:51.093221 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:51.093378 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:51.094334 (+ 956us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:06:51.094354 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:06:51.095935 (+ 1581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:06:51.097160 (+ 1225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:06:51.097166 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:06:51.097167 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:06:51.097336 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:06:51.097443 (+ 107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:06:51.097444 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:06:51.097444 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:06:51.097486 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:06:51.097633 (+ 147us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:06:51.097635 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:06:51.097730 (+ 95us) client_negotiation.cc:769] Sending connection context
0507 02:06:51.097805 (+ 75us) client_negotiation.cc:240] Negotiation successful
0507 02:06:51.097893 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":80,"threads_started":1}
I20240507 02:06:51.098474 1733 heartbeater.cc:347] Connected to a master server at 127.31.131.126:34169
I20240507 02:06:51.098543 1733 heartbeater.cc:464] Registering TS with master...
I20240507 02:06:51.098740 1733 heartbeater.cc:511] Master 127.31.131.126:34169 requested a full tablet report, sending...
I20240507 02:06:51.099225 1528 ts_manager.cc:194] Registered new tserver with Master: eb01f5a060104233b45cb74c075401da (127.31.131.65:33363)
I20240507 02:06:51.099928 1528 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.131.65@KRBTEST.COM'} at 127.31.131.65:47957
I20240507 02:06:51.103322 1578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:51.099382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:41779 (local address 127.31.131.126:34169)
0507 02:06:51.099419 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:51.099421 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:51.099558 (+ 137us) server_negotiation.cc:407] Connection header received
0507 02:06:51.099609 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:51.099610 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:51.099618 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:51.099668 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:06:51.100071 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:51.100515 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:51.101003 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:51.101154 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:51.101660 (+ 506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:06:51.101664 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:06:51.101664 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:06:51.101671 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:06:51.102707 (+ 1036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:51.102934 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:51.102935 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:51.102935 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:51.102950 (+ 15us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:51.103035 (+ 85us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:51.103035 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:51.103035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:51.103099 (+ 64us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:06:51.103137 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:51.103217 (+ 80us) server_negotiation.cc:299] Negotiation successful
0507 02:06:51.103267 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20240507 02:06:51.103331 1739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:51.099387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:34169 (local address 127.31.131.65:41779)
0507 02:06:51.099495 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0507 02:06:51.099503 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0507 02:06:51.099558 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:06:51.099686 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:06:51.099687 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:06:51.099809 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:06:51.099971 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:51.099972 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:51.100579 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:06:51.100580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:06:51.100939 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:51.100940 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:51.100996 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:51.101360 (+ 364us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:06:51.101363 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:06:51.101571 (+ 208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:06:51.102765 (+ 1194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:06:51.102766 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:06:51.102766 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:06:51.102884 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:06:51.102987 (+ 103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:06:51.102987 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:06:51.102987 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:06:51.103007 (+ 20us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:06:51.103132 (+ 125us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:06:51.103133 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:06:51.103161 (+ 28us) client_negotiation.cc:769] Sending connection context
0507 02:06:51.103203 (+ 42us) client_negotiation.cc:240] Negotiation successful
0507 02:06:51.103263 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":66,"threads_started":1}
I20240507 02:06:51.104658 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 1601
I20240507 02:06:51.104743 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/ts-0/wal/instance
I20240507 02:06:51.104863 32269 external_mini_cluster.cc:1428] Setting key 10f0bcbdf00dbee7faa4239ae2eb3b1e
WARNING: no policy specified for kudu/127.31.131.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab.
Entry for principal kudu/127.31.131.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab.
I20240507 02:06:51.124504 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.66:0
--local_ip_for_outbound_sockets=127.31.131.66
--webserver_interface=127.31.131.66
--webserver_port=0
--tserver_master_addrs=127.31.131.126:34169
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab
--principal=kudu/127.31.131.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:41621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:06:51.198570 1742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:06:51.198745 1742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:06:51.198767 1742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:06:51.199998 1742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:06:51.200035 1742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:06:51.200053 1742 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20240507 02:06:51.200063 1742 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20240507 02:06:51.200078 1742 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.66
I20240507 02:06:51.201498 1742 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:41621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab
--principal=kudu/127.31.131.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.131.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.131.126:34169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1742
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.66
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:06:51.201659 1742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:06:51.201813 1742 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240507 02:06:51.205356 1742 server_base.cc:1034] running on GCE node
W20240507 02:06:51.205330 1748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:06:51.205448 1750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:06:51.205319 1747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:06:51.205660 1742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:06:51.205840 1742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:06:51.206962 1742 hybrid_clock.cc:648] HybridClock initialized: now 1715047611206952 us; error 27 us; skew 500 ppm
May 07 02:06:51 dist-test-slave-v0bg krb5kdc[1484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047611, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:06:51.209066 1742 init.cc:377] Logged in from keytab as kudu/127.31.131.66@KRBTEST.COM (short username kudu)
I20240507 02:06:51.209434 1742 webserver.cc:460] Webserver started at http://127.31.131.66:33713/ using document root <none> and password file <none>
I20240507 02:06:51.209589 1742 fs_manager.cc:359] Metadata directory not provided
I20240507 02:06:51.209627 1742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:06:51.209717 1742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:06:51.210624 1742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/ts-1/data/instance:
uuid: "af65885f5957407c938f4af9bc4f7bdc"
format_stamp: "Formatted at 2024-05-07 02:06:51 on dist-test-slave-v0bg"
server_key: "e8f5913e9f7a63852cdeac32c925ea5d"
server_key_iv: "803bf6ac2b1dcdf6cb9a902ed1f4deea"
server_key_version: "encryptionkey@0"
I20240507 02:06:51.210904 1742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/ts-1/wal/instance:
uuid: "af65885f5957407c938f4af9bc4f7bdc"
format_stamp: "Formatted at 2024-05-07 02:06:51 on dist-test-slave-v0bg"
server_key: "e8f5913e9f7a63852cdeac32c925ea5d"
server_key_iv: "803bf6ac2b1dcdf6cb9a902ed1f4deea"
server_key_version: "encryptionkey@0"
I20240507 02:06:51.211911 1742 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240507 02:06:51.212458 1757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:51.212585 1742 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240507 02:06:51.212652 1742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/ts-1/wal
uuid: "af65885f5957407c938f4af9bc4f7bdc"
format_stamp: "Formatted at 2024-05-07 02:06:51 on dist-test-slave-v0bg"
server_key: "e8f5913e9f7a63852cdeac32c925ea5d"
server_key_iv: "803bf6ac2b1dcdf6cb9a902ed1f4deea"
server_key_version: "encryptionkey@0"
I20240507 02:06:51.212707 1742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:06:51.227847 1742 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:06:51.229321 1742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:06:51.229423 1742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:06:51.229897 1742 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:06:51.229933 1742 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:51.229959 1742 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:06:51.229983 1742 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
May 07 02:06:51 dist-test-slave-v0bg krb5kdc[1484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047611, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.66@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:06:51.236340 1742 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.66:43071
I20240507 02:06:51.236407 1873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.66:43071 every 8 connection(s)
I20240507 02:06:51.236685 1742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
I20240507 02:06:51.237578 1578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:51.230228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.66:43449 (local address 127.31.131.126:34169)
0507 02:06:51.230286 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:51.230288 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:51.230436 (+ 148us) server_negotiation.cc:407] Connection header received
0507 02:06:51.231101 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:51.231102 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:51.231116 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:51.231176 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:06:51.231898 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:51.232364 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:51.232884 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:51.233046 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:51.234790 (+ 1744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:06:51.234794 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:06:51.234795 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:06:51.234803 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:06:51.236396 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:51.236811 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:51.236812 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:51.236812 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:51.236835 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:51.237033 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:51.237034 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:51.237035 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:51.237149 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:06:51.237201 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:51.237456 (+ 255us) server_negotiation.cc:299] Negotiation successful
0507 02:06:51.237516 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20240507 02:06:51.237656 1766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:51.230184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:34169 (local address 127.31.131.66:43449)
0507 02:06:51.230323 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0507 02:06:51.230353 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0507 02:06:51.230979 (+ 626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:06:51.231256 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:06:51.231261 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:06:51.231540 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:06:51.231809 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:51.231813 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:51.232435 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:06:51.232436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:06:51.232813 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:51.232814 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:51.232928 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:51.233432 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:06:51.233445 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:06:51.234702 (+ 1257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:06:51.236493 (+ 1791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:06:51.236497 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:06:51.236500 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:06:51.236745 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:06:51.236892 (+ 147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:06:51.236893 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:06:51.236893 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:06:51.236964 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:06:51.237205 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:06:51.237207 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:06:51.237343 (+ 136us) client_negotiation.cc:769] Sending connection context
0507 02:06:51.237429 (+ 86us) client_negotiation.cc:240] Negotiation successful
0507 02:06:51.237524 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":75,"threads_started":1}
I20240507 02:06:51.238000 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 1742
I20240507 02:06:51.238101 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/ts-1/wal/instance
I20240507 02:06:51.238118 1874 heartbeater.cc:347] Connected to a master server at 127.31.131.126:34169
I20240507 02:06:51.238189 1874 heartbeater.cc:464] Registering TS with master...
I20240507 02:06:51.238291 32269 external_mini_cluster.cc:1428] Setting key c2dfbb14b55049af06f48618e30fc077
I20240507 02:06:51.238382 1874 heartbeater.cc:511] Master 127.31.131.126:34169 requested a full tablet report, sending...
I20240507 02:06:51.238837 1528 ts_manager.cc:194] Registered new tserver with Master: af65885f5957407c938f4af9bc4f7bdc (127.31.131.66:43071)
I20240507 02:06:51.239280 1528 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.131.66@KRBTEST.COM'} at 127.31.131.66:43449
I20240507 02:06:51.244580 1578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:51.240065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.66:40043 (local address 127.31.131.126:34169)
0507 02:06:51.240104 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:51.240106 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:51.240229 (+ 123us) server_negotiation.cc:407] Connection header received
0507 02:06:51.240293 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:51.240294 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:51.240304 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:51.240352 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:06:51.240710 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:51.241161 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:51.241864 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:51.241968 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:51.242633 (+ 665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:06:51.242635 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:06:51.242636 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:06:51.242640 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:06:51.243765 (+ 1125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:51.244072 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:51.244073 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:51.244073 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:51.244087 (+ 14us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:51.244216 (+ 129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:51.244216 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:51.244216 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:51.244303 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:06:51.244342 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:51.244475 (+ 133us) server_negotiation.cc:299] Negotiation successful
0507 02:06:51.244523 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20240507 02:06:51.244653 1880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:51.240023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:34169 (local address 127.31.131.66:40043)
0507 02:06:51.240161 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0507 02:06:51.240170 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0507 02:06:51.240234 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:06:51.240365 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:06:51.240366 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:06:51.240483 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:06:51.240648 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:51.240650 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:51.241239 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:06:51.241240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:06:51.241795 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:51.241798 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:51.241869 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:51.242254 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:06:51.242258 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:06:51.242565 (+ 307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:06:51.243841 (+ 1276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:06:51.243843 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:06:51.243843 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:06:51.244009 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:06:51.244136 (+ 127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:06:51.244137 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:06:51.244137 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:06:51.244166 (+ 29us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:06:51.244351 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:06:51.244352 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:06:51.244391 (+ 39us) client_negotiation.cc:769] Sending connection context
0507 02:06:51.244462 (+ 71us) client_negotiation.cc:240] Negotiation successful
0507 02:06:51.244543 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":70,"threads_started":1}
WARNING: no policy specified for kudu/127.31.131.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab.
Entry for principal kudu/127.31.131.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab.
I20240507 02:06:51.258904 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.67:0
--local_ip_for_outbound_sockets=127.31.131.67
--webserver_interface=127.31.131.67
--webserver_port=0
--tserver_master_addrs=127.31.131.126:34169
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab
--principal=kudu/127.31.131.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:41621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:06:51.332557 1883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:06:51.332727 1883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:06:51.332757 1883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:06:51.334015 1883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:06:51.334055 1883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:06:51.334080 1883 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20240507 02:06:51.334095 1883 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20240507 02:06:51.334115 1883 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.67
I20240507 02:06:51.335654 1883 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:41621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab
--principal=kudu/127.31.131.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.131.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.131.126:34169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.67
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:06:51.335830 1883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:06:51.335976 1883 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:06:51.339601 1888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:06:51.339780 1891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:06:51.339752 1889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:06:51.339766 1883 server_base.cc:1034] running on GCE node
I20240507 02:06:51.339998 1883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:06:51.340178 1883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:06:51.341307 1883 hybrid_clock.cc:648] HybridClock initialized: now 1715047611341293 us; error 27 us; skew 500 ppm
May 07 02:06:51 dist-test-slave-v0bg krb5kdc[1484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047611, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:06:51.343555 1883 init.cc:377] Logged in from keytab as kudu/127.31.131.67@KRBTEST.COM (short username kudu)
I20240507 02:06:51.343915 1883 webserver.cc:460] Webserver started at http://127.31.131.67:41775/ using document root <none> and password file <none>
I20240507 02:06:51.344060 1883 fs_manager.cc:359] Metadata directory not provided
I20240507 02:06:51.344098 1883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:06:51.344204 1883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:06:51.345067 1883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/ts-2/data/instance:
uuid: "bc276328e6f24b5aa91552aa9fc95522"
format_stamp: "Formatted at 2024-05-07 02:06:51 on dist-test-slave-v0bg"
server_key: "8c7afef56f444d573b2cbafc1995b703"
server_key_iv: "1e95261bf0aa0f54118aa0329931ff60"
server_key_version: "encryptionkey@0"
I20240507 02:06:51.345341 1883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/ts-2/wal/instance:
uuid: "bc276328e6f24b5aa91552aa9fc95522"
format_stamp: "Formatted at 2024-05-07 02:06:51 on dist-test-slave-v0bg"
server_key: "8c7afef56f444d573b2cbafc1995b703"
server_key_iv: "1e95261bf0aa0f54118aa0329931ff60"
server_key_version: "encryptionkey@0"
I20240507 02:06:51.346393 1883 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240507 02:06:51.347009 1898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:51.347146 1883 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:51.347218 1883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/ts-2/wal
uuid: "bc276328e6f24b5aa91552aa9fc95522"
format_stamp: "Formatted at 2024-05-07 02:06:51 on dist-test-slave-v0bg"
server_key: "8c7afef56f444d573b2cbafc1995b703"
server_key_iv: "1e95261bf0aa0f54118aa0329931ff60"
server_key_version: "encryptionkey@0"
I20240507 02:06:51.347270 1883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:06:51.361408 1883 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:06:51.363197 1883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:06:51.363292 1883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:06:51.363706 1883 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:06:51.363741 1883 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:51.363766 1883 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:06:51.363782 1883 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
May 07 02:06:51 dist-test-slave-v0bg krb5kdc[1484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047611, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.67@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:06:51.369482 1883 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.67:36899
I20240507 02:06:51.369530 2014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.67:36899 every 8 connection(s)
I20240507 02:06:51.369767 1883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
I20240507 02:06:51.372040 1578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:51.364034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.67:38231 (local address 127.31.131.126:34169)
0507 02:06:51.364095 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:51.364097 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:51.364251 (+ 154us) server_negotiation.cc:407] Connection header received
0507 02:06:51.364844 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:51.364845 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:51.364854 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:51.364917 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:06:51.365856 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:51.366631 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:51.367157 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:51.367319 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:51.369879 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:06:51.369883 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:06:51.369883 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:06:51.369889 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:06:51.371033 (+ 1144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:51.371349 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:51.371351 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:51.371351 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:51.371372 (+ 21us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:51.371519 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:51.371519 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:51.371520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:51.371602 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:06:51.371662 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:51.371873 (+ 211us) server_negotiation.cc:299] Negotiation successful
0507 02:06:51.371950 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20240507 02:06:51.372141 1907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:51.364010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:34169 (local address 127.31.131.67:38231)
0507 02:06:51.364140 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0507 02:06:51.364170 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0507 02:06:51.364744 (+ 574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:06:51.364983 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:06:51.364988 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:06:51.365358 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:06:51.365756 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:51.365762 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:51.366718 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:06:51.366719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:06:51.367080 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:51.367082 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:51.367200 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:51.367654 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:06:51.367675 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:06:51.369804 (+ 2129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:06:51.371101 (+ 1297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:06:51.371106 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:06:51.371109 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:06:51.371290 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:06:51.371420 (+ 130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:06:51.371421 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:06:51.371421 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:06:51.371471 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:06:51.371650 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:06:51.371655 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:06:51.371757 (+ 102us) client_negotiation.cc:769] Sending connection context
0507 02:06:51.371844 (+ 87us) client_negotiation.cc:240] Negotiation successful
0507 02:06:51.371954 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":68,"threads_started":1}
I20240507 02:06:51.372185 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 1883
I20240507 02:06:51.372277 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/ts-2/wal/instance
I20240507 02:06:51.372427 32269 external_mini_cluster.cc:1428] Setting key a650d4df456e677d110690d633bf9d29
I20240507 02:06:51.372527 2015 heartbeater.cc:347] Connected to a master server at 127.31.131.126:34169
I20240507 02:06:51.372597 2015 heartbeater.cc:464] Registering TS with master...
I20240507 02:06:51.372785 2015 heartbeater.cc:511] Master 127.31.131.126:34169 requested a full tablet report, sending...
I20240507 02:06:51.373154 1529 ts_manager.cc:194] Registered new tserver with Master: bc276328e6f24b5aa91552aa9fc95522 (127.31.131.67:36899)
I20240507 02:06:51.373601 1529 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.131.67@KRBTEST.COM'} at 127.31.131.67:38231
I20240507 02:06:51.374102 32269 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240507 02:06:51.378059 2021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:51.373490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:34169 (local address 127.31.131.67:41545)
0507 02:06:51.373598 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0507 02:06:51.373607 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0507 02:06:51.373687 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:06:51.373801 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:06:51.373802 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:06:51.373943 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:06:51.374150 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:51.374152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:51.374822 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:06:51.374823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:06:51.375339 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:51.375341 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:51.375405 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:51.375859 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:06:51.375863 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:06:51.376063 (+ 200us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:06:51.377357 (+ 1294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:06:51.377358 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:06:51.377358 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:06:51.377480 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:06:51.377601 (+ 121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:06:51.377601 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:06:51.377601 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:06:51.377627 (+ 26us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:06:51.377787 (+ 160us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:06:51.377788 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:06:51.377824 (+ 36us) client_negotiation.cc:769] Sending connection context
0507 02:06:51.377883 (+ 59us) client_negotiation.cc:240] Negotiation successful
0507 02:06:51.377969 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":56,"threads_started":1}
I20240507 02:06:51.378054 1578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:51.373512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.67:41545 (local address 127.31.131.126:34169)
0507 02:06:51.373549 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:51.373550 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:51.373665 (+ 115us) server_negotiation.cc:407] Connection header received
0507 02:06:51.373737 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:51.373737 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:51.373746 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:51.373794 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:06:51.374244 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:51.374752 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:51.375404 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:51.375506 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:51.376136 (+ 630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:06:51.376138 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:06:51.376139 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:06:51.376144 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:06:51.377289 (+ 1145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:51.377534 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:51.377535 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:51.377535 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:51.377554 (+ 19us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:51.377665 (+ 111us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:51.377666 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:51.377666 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:51.377740 (+ 74us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:06:51.377789 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:51.377909 (+ 120us) server_negotiation.cc:299] Negotiation successful
0507 02:06:51.377969 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
May 07 02:06:51 dist-test-slave-v0bg krb5kdc[1484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047611, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
May 07 02:06:51 dist-test-slave-v0bg krb5kdc[1484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047611, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:06:51.392935 1578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:51.387663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59114 (local address 127.31.131.126:34169)
0507 02:06:51.387704 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:51.387706 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:51.387715 (+ 9us) server_negotiation.cc:407] Connection header received
0507 02:06:51.387753 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:51.387754 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:51.387762 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:51.387811 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:06:51.388416 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:51.388872 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:51.389462 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:51.389558 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:51.391310 (+ 1752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:06:51.391313 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:06:51.391313 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:06:51.391318 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:06:51.392303 (+ 985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:51.392527 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:51.392528 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:51.392528 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:51.392542 (+ 14us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:51.392626 (+ 84us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:51.392626 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:51.392627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:51.392700 (+ 73us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:06:51.392769 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:51.392826 (+ 57us) server_negotiation.cc:299] Negotiation successful
0507 02:06:51.392873 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20240507 02:06:51.394582 1529 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59114:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 02:06:51.396010 1529 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 02:06:51.400568 2029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:51.397684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.65:33363 (local address 127.0.0.1:51460)
0507 02:06:51.397842 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0507 02:06:51.397852 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0507 02:06:51.397908 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:06:51.398091 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:06:51.398094 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:06:51.398096 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:06:51.398314 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:51.398316 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:51.399689 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:06:51.399690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:06:51.400316 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:51.400318 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:51.400376 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:51.400413 (+ 37us) client_negotiation.cc:769] Sending connection context
0507 02:06:51.400460 (+ 47us) client_negotiation.cc:240] Negotiation successful
0507 02:06:51.400497 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":3,"thread_start_us":74,"threads_started":1}
I20240507 02:06:51.401008 2032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:51.397878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.67:36899 (local address 127.0.0.1:47746)
0507 02:06:51.398214 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0507 02:06:51.398223 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0507 02:06:51.398268 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:06:51.398523 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:06:51.398524 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:06:51.398527 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:06:51.398752 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:51.398754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:51.399665 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:06:51.399666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:06:51.400803 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:51.400804 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:51.400863 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:51.400866 (+ 3us) client_negotiation.cc:769] Sending connection context
0507 02:06:51.400895 (+ 29us) client_negotiation.cc:240] Negotiation successful
0507 02:06:51.400931 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":54,"threads_started":1}
I20240507 02:06:51.401065 2031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:51.397781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51460 (local address 127.31.131.65:33363)
0507 02:06:51.397934 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:51.397941 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:51.397952 (+ 11us) server_negotiation.cc:407] Connection header received
0507 02:06:51.397988 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:51.397989 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:51.398003 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:51.398066 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:06:51.398406 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:51.399590 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:51.400416 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:51.400886 (+ 470us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:51.400935 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:51.400963 (+ 28us) server_negotiation.cc:299] Negotiation successful
0507 02:06:51.400999 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":75,"threads_started":1}
I20240507 02:06:51.401010 2030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:51.397684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.66:43071 (local address 127.0.0.1:43084)
0507 02:06:51.397818 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0507 02:06:51.397844 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0507 02:06:51.397909 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:06:51.398352 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:06:51.398353 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:06:51.398355 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:06:51.398597 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:51.398599 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:51.399866 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:06:51.399867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:06:51.400801 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:51.400803 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:51.400865 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:51.400867 (+ 2us) client_negotiation.cc:769] Sending connection context
0507 02:06:51.400895 (+ 28us) client_negotiation.cc:240] Negotiation successful
0507 02:06:51.400931 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":52,"threads_started":1}
I20240507 02:06:51.401597 1667 tablet_service.cc:1467] Processing CreateTablet for tablet 75097fe80cd94d87985f4861c37e675c (DEFAULT_TABLE table=test-table [id=1887317c07a44f3cbb223d7220b66d34]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:06:51.401762 2033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:51.397995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43084 (local address 127.31.131.66:43071)
0507 02:06:51.398153 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:51.398164 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:51.398172 (+ 8us) server_negotiation.cc:407] Connection header received
0507 02:06:51.398216 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:51.398218 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:51.398232 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:51.398343 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:06:51.398675 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:51.399800 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:51.401292 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:51.401615 (+ 323us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:51.401656 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:51.401680 (+ 24us) server_negotiation.cc:299] Negotiation successful
0507 02:06:51.401703 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":77,"threads_started":1}
I20240507 02:06:51.401847 1667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75097fe80cd94d87985f4861c37e675c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:06:51.402287 2034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:51.398214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47746 (local address 127.31.131.67:36899)
0507 02:06:51.398362 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:51.398369 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:51.398377 (+ 8us) server_negotiation.cc:407] Connection header received
0507 02:06:51.398408 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:51.398409 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:51.398423 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:51.398828 (+ 405us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:06:51.398899 (+ 71us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:51.399599 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:51.400894 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:51.402105 (+ 1211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:51.402162 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:51.402193 (+ 31us) server_negotiation.cc:299] Negotiation successful
0507 02:06:51.402228 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":75,"threads_started":1}
I20240507 02:06:51.402249 1808 tablet_service.cc:1467] Processing CreateTablet for tablet 75097fe80cd94d87985f4861c37e675c (DEFAULT_TABLE table=test-table [id=1887317c07a44f3cbb223d7220b66d34]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:06:51.402479 1808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75097fe80cd94d87985f4861c37e675c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:06:51.404711 2035 tablet_bootstrap.cc:492] T 75097fe80cd94d87985f4861c37e675c P eb01f5a060104233b45cb74c075401da: Bootstrap starting.
I20240507 02:06:51.405123 2036 tablet_bootstrap.cc:492] T 75097fe80cd94d87985f4861c37e675c P af65885f5957407c938f4af9bc4f7bdc: Bootstrap starting.
I20240507 02:06:51.405455 2035 tablet_bootstrap.cc:654] T 75097fe80cd94d87985f4861c37e675c P eb01f5a060104233b45cb74c075401da: Neither blocks nor log segments found. Creating new log.
I20240507 02:06:51.405498 1948 tablet_service.cc:1467] Processing CreateTablet for tablet 75097fe80cd94d87985f4861c37e675c (DEFAULT_TABLE table=test-table [id=1887317c07a44f3cbb223d7220b66d34]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:06:51.405742 2035 log.cc:824] T 75097fe80cd94d87985f4861c37e675c P eb01f5a060104233b45cb74c075401da: Log is configured to *not* fsync() on all Append() calls
I20240507 02:06:51.405759 1948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75097fe80cd94d87985f4861c37e675c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:06:51.405835 2036 tablet_bootstrap.cc:654] T 75097fe80cd94d87985f4861c37e675c P af65885f5957407c938f4af9bc4f7bdc: Neither blocks nor log segments found. Creating new log.
I20240507 02:06:51.406103 2036 log.cc:824] T 75097fe80cd94d87985f4861c37e675c P af65885f5957407c938f4af9bc4f7bdc: Log is configured to *not* fsync() on all Append() calls
I20240507 02:06:51.406414 2035 tablet_bootstrap.cc:492] T 75097fe80cd94d87985f4861c37e675c P eb01f5a060104233b45cb74c075401da: No bootstrap required, opened a new log
I20240507 02:06:51.406484 2035 ts_tablet_manager.cc:1367] T 75097fe80cd94d87985f4861c37e675c P eb01f5a060104233b45cb74c075401da: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240507 02:06:51.406834 2036 tablet_bootstrap.cc:492] T 75097fe80cd94d87985f4861c37e675c P af65885f5957407c938f4af9bc4f7bdc: No bootstrap required, opened a new log
I20240507 02:06:51.406903 2036 ts_tablet_manager.cc:1367] T 75097fe80cd94d87985f4861c37e675c P af65885f5957407c938f4af9bc4f7bdc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240507 02:06:51.407707 2035 raft_consensus.cc:348] T 75097fe80cd94d87985f4861c37e675c P eb01f5a060104233b45cb74c075401da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb01f5a060104233b45cb74c075401da" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33363 } } peers { permanent_uuid: "af65885f5957407c938f4af9bc4f7bdc" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 43071 } } peers { permanent_uuid: "bc276328e6f24b5aa91552aa9fc95522" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36899 } }
I20240507 02:06:51.407817 2035 raft_consensus.cc:374] T 75097fe80cd94d87985f4861c37e675c P eb01f5a060104233b45cb74c075401da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:06:51.407837 2035 raft_consensus.cc:729] T 75097fe80cd94d87985f4861c37e675c P eb01f5a060104233b45cb74c075401da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb01f5a060104233b45cb74c075401da, State: Initialized, Role: FOLLOWER
I20240507 02:06:51.407939 2035 consensus_queue.cc:260] T 75097fe80cd94d87985f4861c37e675c P eb01f5a060104233b45cb74c075401da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb01f5a060104233b45cb74c075401da" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33363 } } peers { permanent_uuid: "af65885f5957407c938f4af9bc4f7bdc" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 43071 } } peers { permanent_uuid: "bc276328e6f24b5aa91552aa9fc95522" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36899 } }
I20240507 02:06:51.408140 2035 ts_tablet_manager.cc:1398] T 75097fe80cd94d87985f4861c37e675c P eb01f5a060104233b45cb74c075401da: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240507 02:06:51.408254 1733 heartbeater.cc:503] Master 127.31.131.126:34169 was elected leader, sending a full tablet report...
I20240507 02:06:51.408463 2040 tablet_bootstrap.cc:492] T 75097fe80cd94d87985f4861c37e675c P bc276328e6f24b5aa91552aa9fc95522: Bootstrap starting.
I20240507 02:06:51.408658 2036 raft_consensus.cc:348] T 75097fe80cd94d87985f4861c37e675c P af65885f5957407c938f4af9bc4f7bdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb01f5a060104233b45cb74c075401da" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33363 } } peers { permanent_uuid: "af65885f5957407c938f4af9bc4f7bdc" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 43071 } } peers { permanent_uuid: "bc276328e6f24b5aa91552aa9fc95522" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36899 } }
I20240507 02:06:51.408748 2036 raft_consensus.cc:374] T 75097fe80cd94d87985f4861c37e675c P af65885f5957407c938f4af9bc4f7bdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:06:51.408761 2036 raft_consensus.cc:729] T 75097fe80cd94d87985f4861c37e675c P af65885f5957407c938f4af9bc4f7bdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af65885f5957407c938f4af9bc4f7bdc, State: Initialized, Role: FOLLOWER
I20240507 02:06:51.408816 2036 consensus_queue.cc:260] T 75097fe80cd94d87985f4861c37e675c P af65885f5957407c938f4af9bc4f7bdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb01f5a060104233b45cb74c075401da" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33363 } } peers { permanent_uuid: "af65885f5957407c938f4af9bc4f7bdc" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 43071 } } peers { permanent_uuid: "bc276328e6f24b5aa91552aa9fc95522" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36899 } }
I20240507 02:06:51.408996 2036 ts_tablet_manager.cc:1398] T 75097fe80cd94d87985f4861c37e675c P af65885f5957407c938f4af9bc4f7bdc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240507 02:06:51.409034 1874 heartbeater.cc:503] Master 127.31.131.126:34169 was elected leader, sending a full tablet report...
I20240507 02:06:51.409132 2040 tablet_bootstrap.cc:654] T 75097fe80cd94d87985f4861c37e675c P bc276328e6f24b5aa91552aa9fc95522: Neither blocks nor log segments found. Creating new log.
I20240507 02:06:51.409425 2040 log.cc:824] T 75097fe80cd94d87985f4861c37e675c P bc276328e6f24b5aa91552aa9fc95522: Log is configured to *not* fsync() on all Append() calls
I20240507 02:06:51.410133 2040 tablet_bootstrap.cc:492] T 75097fe80cd94d87985f4861c37e675c P bc276328e6f24b5aa91552aa9fc95522: No bootstrap required, opened a new log
I20240507 02:06:51.410183 2040 ts_tablet_manager.cc:1367] T 75097fe80cd94d87985f4861c37e675c P bc276328e6f24b5aa91552aa9fc95522: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240507 02:06:51.411372 2040 raft_consensus.cc:348] T 75097fe80cd94d87985f4861c37e675c P bc276328e6f24b5aa91552aa9fc95522 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb01f5a060104233b45cb74c075401da" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33363 } } peers { permanent_uuid: "af65885f5957407c938f4af9bc4f7bdc" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 43071 } } peers { permanent_uuid: "bc276328e6f24b5aa91552aa9fc95522" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36899 } }
I20240507 02:06:51.411468 2040 raft_consensus.cc:374] T 75097fe80cd94d87985f4861c37e675c P bc276328e6f24b5aa91552aa9fc95522 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:06:51.411482 2040 raft_consensus.cc:729] T 75097fe80cd94d87985f4861c37e675c P bc276328e6f24b5aa91552aa9fc95522 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc276328e6f24b5aa91552aa9fc95522, State: Initialized, Role: FOLLOWER
I20240507 02:06:51.411547 2040 consensus_queue.cc:260] T 75097fe80cd94d87985f4861c37e675c P bc276328e6f24b5aa91552aa9fc95522 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb01f5a060104233b45cb74c075401da" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33363 } } peers { permanent_uuid: "af65885f5957407c938f4af9bc4f7bdc" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 43071 } } peers { permanent_uuid: "bc276328e6f24b5aa91552aa9fc95522" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36899 } }
I20240507 02:06:51.411741 2040 ts_tablet_manager.cc:1398] T 75097fe80cd94d87985f4861c37e675c P bc276328e6f24b5aa91552aa9fc95522: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240507 02:06:51.411798 2015 heartbeater.cc:503] Master 127.31.131.126:34169 was elected leader, sending a full tablet report...
W20240507 02:06:51.487212 1875 tablet.cc:2369] T 75097fe80cd94d87985f4861c37e675c P af65885f5957407c938f4af9bc4f7bdc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240507 02:06:51.595896 1734 tablet.cc:2369] T 75097fe80cd94d87985f4861c37e675c P eb01f5a060104233b45cb74c075401da: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240507 02:06:51.620263 2016 tablet.cc:2369] T 75097fe80cd94d87985f4861c37e675c P bc276328e6f24b5aa91552aa9fc95522: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240507 02:06:51.654064 2039 raft_consensus.cc:482] T 75097fe80cd94d87985f4861c37e675c P eb01f5a060104233b45cb74c075401da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 02:06:51.654203 2039 raft_consensus.cc:504] T 75097fe80cd94d87985f4861c37e675c P eb01f5a060104233b45cb74c075401da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb01f5a060104233b45cb74c075401da" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33363 } } peers { permanent_uuid: "af65885f5957407c938f4af9bc4f7bdc" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 43071 } } peers { permanent_uuid: "bc276328e6f24b5aa91552aa9fc95522" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36899 } }
I20240507 02:06:51.654517 2039 leader_election.cc:288] T 75097fe80cd94d87985f4861c37e675c P eb01f5a060104233b45cb74c075401da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af65885f5957407c938f4af9bc4f7bdc (127.31.131.66:43071), bc276328e6f24b5aa91552aa9fc95522 (127.31.131.67:36899)
I20240507 02:06:51.657420 2045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:51.654717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.66:43071 (local address 127.31.131.65:59803)
0507 02:06:51.654842 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0507 02:06:51.654851 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0507 02:06:51.654920 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:06:51.655061 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:06:51.655063 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:06:51.655065 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:06:51.655311 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:51.655313 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:51.656232 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:06:51.656233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:06:51.657182 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:51.657186 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:51.657254 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:51.657260 (+ 6us) client_negotiation.cc:769] Sending connection context
0507 02:06:51.657289 (+ 29us) client_negotiation.cc:240] Negotiation successful
0507 02:06:51.657329 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":11,"thread_start_us":49,"threads_started":1}
I20240507 02:06:51.657846 2033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:51.654799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:59803 (local address 127.31.131.66:43071)
0507 02:06:51.654843 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:51.654845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:51.654919 (+ 74us) server_negotiation.cc:407] Connection header received
0507 02:06:51.654997 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:51.654997 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:51.655005 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:51.655050 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:06:51.655376 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:51.656141 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:51.657255 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:51.657754 (+ 499us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:51.657762 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:51.657781 (+ 19us) server_negotiation.cc:299] Negotiation successful
0507 02:06:51.657802 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20240507 02:06:51.657982 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:51.654717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.67:36899 (local address 127.31.131.65:38299)
0507 02:06:51.654839 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0507 02:06:51.654851 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0507 02:06:51.654920 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:06:51.655062 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:06:51.655063 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:06:51.655065 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:06:51.655311 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:51.655313 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:51.656232 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:06:51.656233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:06:51.657264 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:51.657269 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:51.657806 (+ 537us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:51.657810 (+ 4us) client_negotiation.cc:769] Sending connection context
0507 02:06:51.657861 (+ 51us) client_negotiation.cc:240] Negotiation successful
0507 02:06:51.657901 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":58,"threads_started":1}
I20240507 02:06:51.657974 2034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:51.654798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:38299 (local address 127.31.131.67:36899)
0507 02:06:51.654843 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:51.654845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:51.654922 (+ 77us) server_negotiation.cc:407] Connection header received
0507 02:06:51.654979 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:51.654980 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:51.654990 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:51.655050 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:06:51.655376 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:51.656141 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:51.657375 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:51.657775 (+ 400us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:51.657782 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:51.657883 (+ 101us) server_negotiation.cc:299] Negotiation successful
0507 02:06:51.657918 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20240507 02:06:51.658133 1828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75097fe80cd94d87985f4861c37e675c" candidate_uuid: "eb01f5a060104233b45cb74c075401da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af65885f5957407c938f4af9bc4f7bdc" is_pre_election: true
I20240507 02:06:51.658233 1968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75097fe80cd94d87985f4861c37e675c" candidate_uuid: "eb01f5a060104233b45cb74c075401da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc276328e6f24b5aa91552aa9fc95522" is_pre_election: true
I20240507 02:06:51.658278 1828 raft_consensus.cc:2454] T 75097fe80cd94d87985f4861c37e675c P af65885f5957407c938f4af9bc4f7bdc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb01f5a060104233b45cb74c075401da in term 0.
I20240507 02:06:51.658370 1968 raft_consensus.cc:2454] T 75097fe80cd94d87985f4861c37e675c P bc276328e6f24b5aa91552aa9fc95522 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb01f5a060104233b45cb74c075401da in term 0.
I20240507 02:06:51.658473 1618 leader_election.cc:302] T 75097fe80cd94d87985f4861c37e675c P eb01f5a060104233b45cb74c075401da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af65885f5957407c938f4af9bc4f7bdc, eb01f5a060104233b45cb74c075401da; no voters:
I20240507 02:06:51.658609 2039 raft_consensus.cc:2781] T 75097fe80cd94d87985f4861c37e675c P eb01f5a060104233b45cb74c075401da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 02:06:51.658660 2039 raft_consensus.cc:482] T 75097fe80cd94d87985f4861c37e675c P eb01f5a060104233b45cb74c075401da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 02:06:51.658696 2039 raft_consensus.cc:3037] T 75097fe80cd94d87985f4861c37e675c P eb01f5a060104233b45cb74c075401da [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:06:51.659309 2039 raft_consensus.cc:504] T 75097fe80cd94d87985f4861c37e675c P eb01f5a060104233b45cb74c075401da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb01f5a060104233b45cb74c075401da" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33363 } } peers { permanent_uuid: "af65885f5957407c938f4af9bc4f7bdc" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 43071 } } peers { permanent_uuid: "bc276328e6f24b5aa91552aa9fc95522" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36899 } }
I20240507 02:06:51.659428 2039 leader_election.cc:288] T 75097fe80cd94d87985f4861c37e675c P eb01f5a060104233b45cb74c075401da [CANDIDATE]: Term 1 election: Requested vote from peers af65885f5957407c938f4af9bc4f7bdc (127.31.131.66:43071), bc276328e6f24b5aa91552aa9fc95522 (127.31.131.67:36899)
I20240507 02:06:51.659556 1828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75097fe80cd94d87985f4861c37e675c" candidate_uuid: "eb01f5a060104233b45cb74c075401da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af65885f5957407c938f4af9bc4f7bdc"
I20240507 02:06:51.659585 1968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75097fe80cd94d87985f4861c37e675c" candidate_uuid: "eb01f5a060104233b45cb74c075401da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc276328e6f24b5aa91552aa9fc95522"
I20240507 02:06:51.659617 1828 raft_consensus.cc:3037] T 75097fe80cd94d87985f4861c37e675c P af65885f5957407c938f4af9bc4f7bdc [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:06:51.659653 1968 raft_consensus.cc:3037] T 75097fe80cd94d87985f4861c37e675c P bc276328e6f24b5aa91552aa9fc95522 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:06:51.660244 1828 raft_consensus.cc:2454] T 75097fe80cd94d87985f4861c37e675c P af65885f5957407c938f4af9bc4f7bdc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb01f5a060104233b45cb74c075401da in term 1.
I20240507 02:06:51.660305 1968 raft_consensus.cc:2454] T 75097fe80cd94d87985f4861c37e675c P bc276328e6f24b5aa91552aa9fc95522 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb01f5a060104233b45cb74c075401da in term 1.
I20240507 02:06:51.660387 1618 leader_election.cc:302] T 75097fe80cd94d87985f4861c37e675c P eb01f5a060104233b45cb74c075401da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af65885f5957407c938f4af9bc4f7bdc, eb01f5a060104233b45cb74c075401da; no voters:
I20240507 02:06:51.660463 2039 raft_consensus.cc:2781] T 75097fe80cd94d87985f4861c37e675c P eb01f5a060104233b45cb74c075401da [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:06:51.660550 2039 raft_consensus.cc:686] T 75097fe80cd94d87985f4861c37e675c P eb01f5a060104233b45cb74c075401da [term 1 LEADER]: Becoming Leader. State: Replica: eb01f5a060104233b45cb74c075401da, State: Running, Role: LEADER
I20240507 02:06:51.660648 2039 consensus_queue.cc:237] T 75097fe80cd94d87985f4861c37e675c P eb01f5a060104233b45cb74c075401da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb01f5a060104233b45cb74c075401da" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33363 } } peers { permanent_uuid: "af65885f5957407c938f4af9bc4f7bdc" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 43071 } } peers { permanent_uuid: "bc276328e6f24b5aa91552aa9fc95522" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36899 } }
I20240507 02:06:51.661346 1528 catalog_manager.cc:5484] T 75097fe80cd94d87985f4861c37e675c P eb01f5a060104233b45cb74c075401da reported cstate change: term changed from 0 to 1, leader changed from <none> to eb01f5a060104233b45cb74c075401da (127.31.131.65). New cstate: current_term: 1 leader_uuid: "eb01f5a060104233b45cb74c075401da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb01f5a060104233b45cb74c075401da" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33363 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af65885f5957407c938f4af9bc4f7bdc" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 43071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc276328e6f24b5aa91552aa9fc95522" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36899 } health_report { overall_health: UNKNOWN } } }
May 07 02:06:51 dist-test-slave-v0bg krb5kdc[1484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047610, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.126@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:06:51.708802 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:51.703259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:34169 (local address 127.0.0.1:59124)
0507 02:06:51.703407 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0507 02:06:51.703415 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0507 02:06:51.703481 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:06:51.703593 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:06:51.703594 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:06:51.703942 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:06:51.704098 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:51.704101 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:51.704702 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:06:51.704703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:06:51.705044 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:51.705045 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:51.705118 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:51.705613 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:06:51.705617 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:06:51.706861 (+ 1244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:06:51.707986 (+ 1125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:06:51.707991 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:06:51.707994 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:06:51.708153 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:06:51.708278 (+ 125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:06:51.708278 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:06:51.708279 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:06:51.708302 (+ 23us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:06:51.708467 (+ 165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:06:51.708473 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:06:51.708513 (+ 40us) client_negotiation.cc:769] Sending connection context
0507 02:06:51.708564 (+ 51us) client_negotiation.cc:240] Negotiation successful
0507 02:06:51.708672 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":98,"threads_started":1}
I20240507 02:06:51.708814 1578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:51.703308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59124 (local address 127.31.131.126:34169)
0507 02:06:51.703372 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:51.703374 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:51.703477 (+ 103us) server_negotiation.cc:407] Connection header received
0507 02:06:51.703528 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:51.703529 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:51.703538 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:51.703584 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:06:51.704184 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:51.704638 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:51.705115 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:51.705266 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:51.706937 (+ 1671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:06:51.706940 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:06:51.706940 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:06:51.706945 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:06:51.707920 (+ 975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:51.708210 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:51.708211 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:51.708212 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:51.708231 (+ 19us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:51.708344 (+ 113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:51.708344 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:51.708344 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:51.708420 (+ 76us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:06:51.708470 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:51.708592 (+ 122us) server_negotiation.cc:299] Negotiation successful
0507 02:06:51.708731 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":76,"server-negotiator.queue_time_us":37}
I20240507 02:06:51.710619 1528 catalog_manager.cc:1904] Servicing CreateTable request from {username='kudu', principal='kudu/127.31.131.126@KRBTEST.COM'} at 127.0.0.1:59124:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20240507 02:06:51.710805 1528 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 02:06:51.712152 1667 tablet_service.cc:1467] Processing CreateTablet for tablet 6f0843ee34b34b3ebe2f0415c44f622f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a07d394160b64bd586c7a9f3d2200c5b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20240507 02:06:51.712179 1808 tablet_service.cc:1467] Processing CreateTablet for tablet 6f0843ee34b34b3ebe2f0415c44f622f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a07d394160b64bd586c7a9f3d2200c5b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20240507 02:06:51.712287 1667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f0843ee34b34b3ebe2f0415c44f622f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:06:51.712306 1808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f0843ee34b34b3ebe2f0415c44f622f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:06:51.713323 2035 tablet_bootstrap.cc:492] T 6f0843ee34b34b3ebe2f0415c44f622f P eb01f5a060104233b45cb74c075401da: Bootstrap starting.
I20240507 02:06:51.713449 1948 tablet_service.cc:1467] Processing CreateTablet for tablet 6f0843ee34b34b3ebe2f0415c44f622f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a07d394160b64bd586c7a9f3d2200c5b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20240507 02:06:51.713577 1948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f0843ee34b34b3ebe2f0415c44f622f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:06:51.713701 2035 tablet_bootstrap.cc:654] T 6f0843ee34b34b3ebe2f0415c44f622f P eb01f5a060104233b45cb74c075401da: Neither blocks nor log segments found. Creating new log.
I20240507 02:06:51.714325 2035 tablet_bootstrap.cc:492] T 6f0843ee34b34b3ebe2f0415c44f622f P eb01f5a060104233b45cb74c075401da: No bootstrap required, opened a new log
I20240507 02:06:51.714368 2035 ts_tablet_manager.cc:1367] T 6f0843ee34b34b3ebe2f0415c44f622f P eb01f5a060104233b45cb74c075401da: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20240507 02:06:51.714521 2035 raft_consensus.cc:348] T 6f0843ee34b34b3ebe2f0415c44f622f P eb01f5a060104233b45cb74c075401da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb01f5a060104233b45cb74c075401da" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33363 } } peers { permanent_uuid: "af65885f5957407c938f4af9bc4f7bdc" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 43071 } } peers { permanent_uuid: "bc276328e6f24b5aa91552aa9fc95522" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36899 } }
I20240507 02:06:51.714598 2035 raft_consensus.cc:374] T 6f0843ee34b34b3ebe2f0415c44f622f P eb01f5a060104233b45cb74c075401da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:06:51.714614 2035 raft_consensus.cc:729] T 6f0843ee34b34b3ebe2f0415c44f622f P eb01f5a060104233b45cb74c075401da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb01f5a060104233b45cb74c075401da, State: Initialized, Role: FOLLOWER
I20240507 02:06:51.714660 2035 consensus_queue.cc:260] T 6f0843ee34b34b3ebe2f0415c44f622f P eb01f5a060104233b45cb74c075401da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb01f5a060104233b45cb74c075401da" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33363 } } peers { permanent_uuid: "af65885f5957407c938f4af9bc4f7bdc" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 43071 } } peers { permanent_uuid: "bc276328e6f24b5aa91552aa9fc95522" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36899 } }
I20240507 02:06:51.714708 2040 tablet_bootstrap.cc:492] T 6f0843ee34b34b3ebe2f0415c44f622f P bc276328e6f24b5aa91552aa9fc95522: Bootstrap starting.
I20240507 02:06:51.714747 2035 ts_tablet_manager.cc:1398] T 6f0843ee34b34b3ebe2f0415c44f622f P eb01f5a060104233b45cb74c075401da: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:51.714746 2036 tablet_bootstrap.cc:492] T 6f0843ee34b34b3ebe2f0415c44f622f P af65885f5957407c938f4af9bc4f7bdc: Bootstrap starting.
I20240507 02:06:51.715080 2036 tablet_bootstrap.cc:654] T 6f0843ee34b34b3ebe2f0415c44f622f P af65885f5957407c938f4af9bc4f7bdc: Neither blocks nor log segments found. Creating new log.
I20240507 02:06:51.715080 2040 tablet_bootstrap.cc:654] T 6f0843ee34b34b3ebe2f0415c44f622f P bc276328e6f24b5aa91552aa9fc95522: Neither blocks nor log segments found. Creating new log.
I20240507 02:06:51.715581 2039 tablet_replica.cc:439] T 6f0843ee34b34b3ebe2f0415c44f622f P eb01f5a060104233b45cb74c075401da: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb01f5a060104233b45cb74c075401da" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33363 } } peers { permanent_uuid: "af65885f5957407c938f4af9bc4f7bdc" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 43071 } } peers { permanent_uuid: "bc276328e6f24b5aa91552aa9fc95522" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36899 } } }
I20240507 02:06:51.715662 2039 tablet_replica.cc:442] T 6f0843ee34b34b3ebe2f0415c44f622f P eb01f5a060104233b45cb74c075401da: This TxnStatusTablet replica's current role is: FOLLOWER
I20240507 02:06:51.716131 2036 tablet_bootstrap.cc:492] T 6f0843ee34b34b3ebe2f0415c44f622f P af65885f5957407c938f4af9bc4f7bdc: No bootstrap required, opened a new log
I20240507 02:06:51.716131 2040 tablet_bootstrap.cc:492] T 6f0843ee34b34b3ebe2f0415c44f622f P bc276328e6f24b5aa91552aa9fc95522: No bootstrap required, opened a new log
I20240507 02:06:51.716179 2040 ts_tablet_manager.cc:1367] T 6f0843ee34b34b3ebe2f0415c44f622f P bc276328e6f24b5aa91552aa9fc95522: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240507 02:06:51.716179 2036 ts_tablet_manager.cc:1367] T 6f0843ee34b34b3ebe2f0415c44f622f P af65885f5957407c938f4af9bc4f7bdc: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240507 02:06:51.716323 2040 raft_consensus.cc:348] T 6f0843ee34b34b3ebe2f0415c44f622f P bc276328e6f24b5aa91552aa9fc95522 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb01f5a060104233b45cb74c075401da" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33363 } } peers { permanent_uuid: "af65885f5957407c938f4af9bc4f7bdc" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 43071 } } peers { permanent_uuid: "bc276328e6f24b5aa91552aa9fc95522" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36899 } }
I20240507 02:06:51.716331 2036 raft_consensus.cc:348] T 6f0843ee34b34b3ebe2f0415c44f622f P af65885f5957407c938f4af9bc4f7bdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb01f5a060104233b45cb74c075401da" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33363 } } peers { permanent_uuid: "af65885f5957407c938f4af9bc4f7bdc" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 43071 } } peers { permanent_uuid: "bc276328e6f24b5aa91552aa9fc95522" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36899 } }
I20240507 02:06:51.716388 2036 raft_consensus.cc:374] T 6f0843ee34b34b3ebe2f0415c44f622f P af65885f5957407c938f4af9bc4f7bdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:06:51.716388 2040 raft_consensus.cc:374] T 6f0843ee34b34b3ebe2f0415c44f622f P bc276328e6f24b5aa91552aa9fc95522 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:06:51.716408 2040 raft_consensus.cc:729] T 6f0843ee34b34b3ebe2f0415c44f622f P bc276328e6f24b5aa91552aa9fc95522 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc276328e6f24b5aa91552aa9fc95522, State: Initialized, Role: FOLLOWER
I20240507 02:06:51.716408 2036 raft_consensus.cc:729] T 6f0843ee34b34b3ebe2f0415c44f622f P af65885f5957407c938f4af9bc4f7bdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af65885f5957407c938f4af9bc4f7bdc, State: Initialized, Role: FOLLOWER
I20240507 02:06:51.716454 2036 consensus_queue.cc:260] T 6f0843ee34b34b3ebe2f0415c44f622f P af65885f5957407c938f4af9bc4f7bdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb01f5a060104233b45cb74c075401da" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33363 } } peers { permanent_uuid: "af65885f5957407c938f4af9bc4f7bdc" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 43071 } } peers { permanent_uuid: "bc276328e6f24b5aa91552aa9fc95522" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36899 } }
I20240507 02:06:51.716451 2040 consensus_queue.cc:260] T 6f0843ee34b34b3ebe2f0415c44f622f P bc276328e6f24b5aa91552aa9fc95522 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb01f5a060104233b45cb74c075401da" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33363 } } peers { permanent_uuid: "af65885f5957407c938f4af9bc4f7bdc" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 43071 } } peers { permanent_uuid: "bc276328e6f24b5aa91552aa9fc95522" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36899 } }
I20240507 02:06:51.716536 2036 ts_tablet_manager.cc:1398] T 6f0843ee34b34b3ebe2f0415c44f622f P af65885f5957407c938f4af9bc4f7bdc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:51.716536 2040 ts_tablet_manager.cc:1398] T 6f0843ee34b34b3ebe2f0415c44f622f P bc276328e6f24b5aa91552aa9fc95522: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20240507 02:06:51.716635 2043 tablet_replica.cc:439] T 6f0843ee34b34b3ebe2f0415c44f622f P bc276328e6f24b5aa91552aa9fc95522: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb01f5a060104233b45cb74c075401da" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33363 } } peers { permanent_uuid: "af65885f5957407c938f4af9bc4f7bdc" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 43071 } } peers { permanent_uuid: "bc276328e6f24b5aa91552aa9fc95522" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36899 } } }
I20240507 02:06:51.716607 2041 tablet_replica.cc:439] T 6f0843ee34b34b3ebe2f0415c44f622f P af65885f5957407c938f4af9bc4f7bdc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb01f5a060104233b45cb74c075401da" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33363 } } peers { permanent_uuid: "af65885f5957407c938f4af9bc4f7bdc" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 43071 } } peers { permanent_uuid: "bc276328e6f24b5aa91552aa9fc95522" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36899 } } }
I20240507 02:06:51.716711 2043 tablet_replica.cc:442] T 6f0843ee34b34b3ebe2f0415c44f622f P bc276328e6f24b5aa91552aa9fc95522: This TxnStatusTablet replica's current role is: FOLLOWER
I20240507 02:06:51.716713 2041 tablet_replica.cc:442] T 6f0843ee34b34b3ebe2f0415c44f622f P af65885f5957407c938f4af9bc4f7bdc: This TxnStatusTablet replica's current role is: FOLLOWER
I20240507 02:06:51.823521 2039 raft_consensus.cc:482] T 6f0843ee34b34b3ebe2f0415c44f622f P eb01f5a060104233b45cb74c075401da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 02:06:51.823614 2039 raft_consensus.cc:504] T 6f0843ee34b34b3ebe2f0415c44f622f P eb01f5a060104233b45cb74c075401da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb01f5a060104233b45cb74c075401da" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33363 } } peers { permanent_uuid: "af65885f5957407c938f4af9bc4f7bdc" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 43071 } } peers { permanent_uuid: "bc276328e6f24b5aa91552aa9fc95522" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36899 } }
I20240507 02:06:51.823783 2039 leader_election.cc:288] T 6f0843ee34b34b3ebe2f0415c44f622f P eb01f5a060104233b45cb74c075401da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af65885f5957407c938f4af9bc4f7bdc (127.31.131.66:43071), bc276328e6f24b5aa91552aa9fc95522 (127.31.131.67:36899)
I20240507 02:06:51.823979 1828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f0843ee34b34b3ebe2f0415c44f622f" candidate_uuid: "eb01f5a060104233b45cb74c075401da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af65885f5957407c938f4af9bc4f7bdc" is_pre_election: true
I20240507 02:06:51.824000 1968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f0843ee34b34b3ebe2f0415c44f622f" candidate_uuid: "eb01f5a060104233b45cb74c075401da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc276328e6f24b5aa91552aa9fc95522" is_pre_election: true
I20240507 02:06:51.824062 1828 raft_consensus.cc:2454] T 6f0843ee34b34b3ebe2f0415c44f622f P af65885f5957407c938f4af9bc4f7bdc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb01f5a060104233b45cb74c075401da in term 0.
I20240507 02:06:51.824093 1968 raft_consensus.cc:2454] T 6f0843ee34b34b3ebe2f0415c44f622f P bc276328e6f24b5aa91552aa9fc95522 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb01f5a060104233b45cb74c075401da in term 0.
I20240507 02:06:51.824187 1618 leader_election.cc:302] T 6f0843ee34b34b3ebe2f0415c44f622f P eb01f5a060104233b45cb74c075401da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af65885f5957407c938f4af9bc4f7bdc, eb01f5a060104233b45cb74c075401da; no voters:
I20240507 02:06:51.824290 2039 raft_consensus.cc:2781] T 6f0843ee34b34b3ebe2f0415c44f622f P eb01f5a060104233b45cb74c075401da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 02:06:51.824334 2039 raft_consensus.cc:482] T 6f0843ee34b34b3ebe2f0415c44f622f P eb01f5a060104233b45cb74c075401da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 02:06:51.824347 2039 raft_consensus.cc:3037] T 6f0843ee34b34b3ebe2f0415c44f622f P eb01f5a060104233b45cb74c075401da [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:06:51.824947 2039 raft_consensus.cc:504] T 6f0843ee34b34b3ebe2f0415c44f622f P eb01f5a060104233b45cb74c075401da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb01f5a060104233b45cb74c075401da" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33363 } } peers { permanent_uuid: "af65885f5957407c938f4af9bc4f7bdc" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 43071 } } peers { permanent_uuid: "bc276328e6f24b5aa91552aa9fc95522" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36899 } }
I20240507 02:06:51.825057 2039 leader_election.cc:288] T 6f0843ee34b34b3ebe2f0415c44f622f P eb01f5a060104233b45cb74c075401da [CANDIDATE]: Term 1 election: Requested vote from peers af65885f5957407c938f4af9bc4f7bdc (127.31.131.66:43071), bc276328e6f24b5aa91552aa9fc95522 (127.31.131.67:36899)
I20240507 02:06:51.825220 1828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f0843ee34b34b3ebe2f0415c44f622f" candidate_uuid: "eb01f5a060104233b45cb74c075401da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af65885f5957407c938f4af9bc4f7bdc"
I20240507 02:06:51.825244 1968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f0843ee34b34b3ebe2f0415c44f622f" candidate_uuid: "eb01f5a060104233b45cb74c075401da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc276328e6f24b5aa91552aa9fc95522"
I20240507 02:06:51.825286 1828 raft_consensus.cc:3037] T 6f0843ee34b34b3ebe2f0415c44f622f P af65885f5957407c938f4af9bc4f7bdc [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:06:51.825300 1968 raft_consensus.cc:3037] T 6f0843ee34b34b3ebe2f0415c44f622f P bc276328e6f24b5aa91552aa9fc95522 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:06:51.825846 1968 raft_consensus.cc:2454] T 6f0843ee34b34b3ebe2f0415c44f622f P bc276328e6f24b5aa91552aa9fc95522 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb01f5a060104233b45cb74c075401da in term 1.
I20240507 02:06:51.825863 1828 raft_consensus.cc:2454] T 6f0843ee34b34b3ebe2f0415c44f622f P af65885f5957407c938f4af9bc4f7bdc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb01f5a060104233b45cb74c075401da in term 1.
I20240507 02:06:51.826036 1621 leader_election.cc:302] T 6f0843ee34b34b3ebe2f0415c44f622f P eb01f5a060104233b45cb74c075401da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc276328e6f24b5aa91552aa9fc95522, eb01f5a060104233b45cb74c075401da; no voters:
I20240507 02:06:51.826469 2039 raft_consensus.cc:2781] T 6f0843ee34b34b3ebe2f0415c44f622f P eb01f5a060104233b45cb74c075401da [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:06:51.826521 2039 raft_consensus.cc:686] T 6f0843ee34b34b3ebe2f0415c44f622f P eb01f5a060104233b45cb74c075401da [term 1 LEADER]: Becoming Leader. State: Replica: eb01f5a060104233b45cb74c075401da, State: Running, Role: LEADER
I20240507 02:06:51.826581 2039 consensus_queue.cc:237] T 6f0843ee34b34b3ebe2f0415c44f622f P eb01f5a060104233b45cb74c075401da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb01f5a060104233b45cb74c075401da" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33363 } } peers { permanent_uuid: "af65885f5957407c938f4af9bc4f7bdc" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 43071 } } peers { permanent_uuid: "bc276328e6f24b5aa91552aa9fc95522" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36899 } }
I20240507 02:06:51.826866 2046 tablet_replica.cc:439] T 6f0843ee34b34b3ebe2f0415c44f622f P eb01f5a060104233b45cb74c075401da: TxnStatusTablet state changed. Reason: New leader eb01f5a060104233b45cb74c075401da. Latest consensus state: current_term: 1 leader_uuid: "eb01f5a060104233b45cb74c075401da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb01f5a060104233b45cb74c075401da" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33363 } } peers { permanent_uuid: "af65885f5957407c938f4af9bc4f7bdc" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 43071 } } peers { permanent_uuid: "bc276328e6f24b5aa91552aa9fc95522" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36899 } } }
I20240507 02:06:51.826956 2046 tablet_replica.cc:442] T 6f0843ee34b34b3ebe2f0415c44f622f P eb01f5a060104233b45cb74c075401da: This TxnStatusTablet replica's current role is: LEADER
I20240507 02:06:51.827138 2058 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20240507 02:06:51.827131 1529 catalog_manager.cc:5484] T 6f0843ee34b34b3ebe2f0415c44f622f P eb01f5a060104233b45cb74c075401da reported cstate change: term changed from 0 to 1, leader changed from <none> to eb01f5a060104233b45cb74c075401da (127.31.131.65). New cstate: current_term: 1 leader_uuid: "eb01f5a060104233b45cb74c075401da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb01f5a060104233b45cb74c075401da" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33363 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af65885f5957407c938f4af9bc4f7bdc" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 43071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc276328e6f24b5aa91552aa9fc95522" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36899 } health_report { overall_health: UNKNOWN } } }
I20240507 02:06:51.827244 2058 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20240507 02:06:51.859506 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:51.857137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.65:33363 (local address 127.0.0.1:58892)
0507 02:06:51.857220 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0507 02:06:51.857231 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0507 02:06:51.857276 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:06:51.857439 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:06:51.857439 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:06:51.857440 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0507 02:06:51.857589 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:51.857590 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:51.858492 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:06:51.858493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:06:51.858924 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:51.858926 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:51.858985 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:51.858997 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0507 02:06:51.859391 (+ 394us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0507 02:06:51.859395 (+ 4us) client_negotiation.cc:769] Sending connection context
0507 02:06:51.859414 (+ 19us) client_negotiation.cc:240] Negotiation successful
0507 02:06:51.859447 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20240507 02:06:51.859522 2031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:51.857209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58892 (local address 127.31.131.65:33363)
0507 02:06:51.857258 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:51.857260 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:51.857270 (+ 10us) server_negotiation.cc:407] Connection header received
0507 02:06:51.857346 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:51.857347 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:51.857365 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:51.857438 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0507 02:06:51.857651 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:51.858430 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:51.858999 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:51.859184 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:51.859223 (+ 39us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0507 02:06:51.859330 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0507 02:06:51.859403 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:51.859428 (+ 25us) server_negotiation.cc:299] Negotiation successful
0507 02:06:51.859451 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20240507 02:06:51.861253 1968 raft_consensus.cc:1261] T 6f0843ee34b34b3ebe2f0415c44f622f P bc276328e6f24b5aa91552aa9fc95522 [term 1 FOLLOWER]: Refusing update from remote peer eb01f5a060104233b45cb74c075401da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 02:06:51.861281 1828 raft_consensus.cc:1261] T 6f0843ee34b34b3ebe2f0415c44f622f P af65885f5957407c938f4af9bc4f7bdc [term 1 FOLLOWER]: Refusing update from remote peer eb01f5a060104233b45cb74c075401da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 02:06:51.861567 2046 consensus_queue.cc:1035] T 6f0843ee34b34b3ebe2f0415c44f622f P eb01f5a060104233b45cb74c075401da [LEADER]: Connected to new peer: Peer: permanent_uuid: "af65885f5957407c938f4af9bc4f7bdc" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 43071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 02:06:51.861694 2039 consensus_queue.cc:1035] T 6f0843ee34b34b3ebe2f0415c44f622f P eb01f5a060104233b45cb74c075401da [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc276328e6f24b5aa91552aa9fc95522" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 02:06:51.862833 2041 tablet_replica.cc:439] T 6f0843ee34b34b3ebe2f0415c44f622f P af65885f5957407c938f4af9bc4f7bdc: TxnStatusTablet state changed. Reason: New leader eb01f5a060104233b45cb74c075401da. Latest consensus state: current_term: 1 leader_uuid: "eb01f5a060104233b45cb74c075401da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb01f5a060104233b45cb74c075401da" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33363 } } peers { permanent_uuid: "af65885f5957407c938f4af9bc4f7bdc" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 43071 } } peers { permanent_uuid: "bc276328e6f24b5aa91552aa9fc95522" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36899 } } }
I20240507 02:06:51.862915 2041 tablet_replica.cc:442] T 6f0843ee34b34b3ebe2f0415c44f622f P af65885f5957407c938f4af9bc4f7bdc: This TxnStatusTablet replica's current role is: FOLLOWER
I20240507 02:06:51.862907 2043 tablet_replica.cc:439] T 6f0843ee34b34b3ebe2f0415c44f622f P bc276328e6f24b5aa91552aa9fc95522: TxnStatusTablet state changed. Reason: New leader eb01f5a060104233b45cb74c075401da. Latest consensus state: current_term: 1 leader_uuid: "eb01f5a060104233b45cb74c075401da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb01f5a060104233b45cb74c075401da" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33363 } } peers { permanent_uuid: "af65885f5957407c938f4af9bc4f7bdc" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 43071 } } peers { permanent_uuid: "bc276328e6f24b5aa91552aa9fc95522" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36899 } } }
I20240507 02:06:51.862983 2043 tablet_replica.cc:442] T 6f0843ee34b34b3ebe2f0415c44f622f P bc276328e6f24b5aa91552aa9fc95522: This TxnStatusTablet replica's current role is: FOLLOWER
I20240507 02:06:51.863863 2046 tablet_replica.cc:439] T 6f0843ee34b34b3ebe2f0415c44f622f P eb01f5a060104233b45cb74c075401da: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eb01f5a060104233b45cb74c075401da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb01f5a060104233b45cb74c075401da" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33363 } } peers { permanent_uuid: "af65885f5957407c938f4af9bc4f7bdc" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 43071 } } peers { permanent_uuid: "bc276328e6f24b5aa91552aa9fc95522" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36899 } } }
I20240507 02:06:51.863946 2046 tablet_replica.cc:442] T 6f0843ee34b34b3ebe2f0415c44f622f P eb01f5a060104233b45cb74c075401da: This TxnStatusTablet replica's current role is: LEADER
I20240507 02:06:51.864039 2043 tablet_replica.cc:439] T 6f0843ee34b34b3ebe2f0415c44f622f P bc276328e6f24b5aa91552aa9fc95522: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eb01f5a060104233b45cb74c075401da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb01f5a060104233b45cb74c075401da" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33363 } } peers { permanent_uuid: "af65885f5957407c938f4af9bc4f7bdc" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 43071 } } peers { permanent_uuid: "bc276328e6f24b5aa91552aa9fc95522" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36899 } } }
I20240507 02:06:51.864111 2043 tablet_replica.cc:442] T 6f0843ee34b34b3ebe2f0415c44f622f P bc276328e6f24b5aa91552aa9fc95522: This TxnStatusTablet replica's current role is: FOLLOWER
I20240507 02:06:51.864277 2063 mvcc.cc:205] Tried to move back new op lower bound from 7024835018181513216 to 7024835018042048512. Current Snapshot: MvccSnapshot[applied={T|T < 7024835018181513216}]
I20240507 02:06:51.864668 2046 tablet_replica.cc:439] T 6f0843ee34b34b3ebe2f0415c44f622f P eb01f5a060104233b45cb74c075401da: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eb01f5a060104233b45cb74c075401da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb01f5a060104233b45cb74c075401da" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33363 } } peers { permanent_uuid: "af65885f5957407c938f4af9bc4f7bdc" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 43071 } } peers { permanent_uuid: "bc276328e6f24b5aa91552aa9fc95522" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36899 } } }
I20240507 02:06:51.864742 2046 tablet_replica.cc:442] T 6f0843ee34b34b3ebe2f0415c44f622f P eb01f5a060104233b45cb74c075401da: This TxnStatusTablet replica's current role is: LEADER
I20240507 02:06:51.865129 2062 mvcc.cc:205] Tried to move back new op lower bound from 7024835018181513216 to 7024835018042048512. Current Snapshot: MvccSnapshot[applied={T|T < 7024835018181513216}]
I20240507 02:06:51.865164 2041 tablet_replica.cc:439] T 6f0843ee34b34b3ebe2f0415c44f622f P af65885f5957407c938f4af9bc4f7bdc: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eb01f5a060104233b45cb74c075401da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb01f5a060104233b45cb74c075401da" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33363 } } peers { permanent_uuid: "af65885f5957407c938f4af9bc4f7bdc" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 43071 } } peers { permanent_uuid: "bc276328e6f24b5aa91552aa9fc95522" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36899 } } }
I20240507 02:06:51.865227 2041 tablet_replica.cc:442] T 6f0843ee34b34b3ebe2f0415c44f622f P af65885f5957407c938f4af9bc4f7bdc: This TxnStatusTablet replica's current role is: FOLLOWER
I20240507 02:06:51.869115 2031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:51.867172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58898 (local address 127.31.131.65:33363)
0507 02:06:51.867221 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:51.867223 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:51.867231 (+ 8us) server_negotiation.cc:407] Connection header received
0507 02:06:51.867250 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:51.867251 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:51.867260 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:51.867308 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0507 02:06:51.867509 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:51.868217 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:51.868711 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:51.868875 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:51.868893 (+ 18us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0507 02:06:51.868930 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0507 02:06:51.868979 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:51.869032 (+ 53us) server_negotiation.cc:299] Negotiation successful
0507 02:06:51.869052 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20240507 02:06:51.872924 2031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:51.871127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:38283 (local address 127.31.131.65:33363)
0507 02:06:51.871168 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:51.871170 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:51.871267 (+ 97us) server_negotiation.cc:407] Connection header received
0507 02:06:51.871320 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:51.871321 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:51.871328 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:51.871370 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0507 02:06:51.871590 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:51.872020 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:51.872583 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:51.872710 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:51.872725 (+ 15us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0507 02:06:51.872742 (+ 17us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0507 02:06:51.872783 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:51.872842 (+ 59us) server_negotiation.cc:299] Negotiation successful
0507 02:06:51.872857 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20240507 02:06:51.872949 2073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:51.871072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.65:33363 (local address 127.31.131.65:38283)
0507 02:06:51.871203 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0507 02:06:51.871212 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0507 02:06:51.871271 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:06:51.871376 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:06:51.871376 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:06:51.871378 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0507 02:06:51.871539 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:51.871540 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:51.872079 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:06:51.872080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:06:51.872517 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:51.872519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:51.872577 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:51.872592 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0507 02:06:51.872794 (+ 202us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0507 02:06:51.872795 (+ 1us) client_negotiation.cc:769] Sending connection context
0507 02:06:51.872835 (+ 40us) client_negotiation.cc:240] Negotiation successful
0507 02:06:51.872869 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":62,"threads_started":1}
I20240507 02:06:51.874855 1828 raft_consensus.cc:1261] T 75097fe80cd94d87985f4861c37e675c P af65885f5957407c938f4af9bc4f7bdc [term 1 FOLLOWER]: Refusing update from remote peer eb01f5a060104233b45cb74c075401da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 02:06:51.874981 1968 raft_consensus.cc:1261] T 75097fe80cd94d87985f4861c37e675c P bc276328e6f24b5aa91552aa9fc95522 [term 1 FOLLOWER]: Refusing update from remote peer eb01f5a060104233b45cb74c075401da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 02:06:51.875193 2068 consensus_queue.cc:1035] T 75097fe80cd94d87985f4861c37e675c P eb01f5a060104233b45cb74c075401da [LEADER]: Connected to new peer: Peer: permanent_uuid: "af65885f5957407c938f4af9bc4f7bdc" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 43071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 02:06:51.875317 2039 consensus_queue.cc:1035] T 75097fe80cd94d87985f4861c37e675c P eb01f5a060104233b45cb74c075401da [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc276328e6f24b5aa91552aa9fc95522" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 02:06:51.876322 2061 mvcc.cc:205] Tried to move back new op lower bound from 7024835018238296064 to 7024835017362563072. Current Snapshot: MvccSnapshot[applied={T|T < 7024835018238296064}]
I20240507 02:06:51.886019 1529 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59114:
table { table_name: "test-table" } modify_external_catalogs: true
I20240507 02:06:51.886102 1529 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59114:
table { table_name: "test-table" } modify_external_catalogs: true
I20240507 02:06:51.886766 1529 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 27251517e7da4be8bebf6abfb08c2a57: Sending DeleteTablet for 3 replicas of tablet 75097fe80cd94d87985f4861c37e675c
I20240507 02:06:51.887027 1948 tablet_service.cc:1514] Processing DeleteTablet for tablet 75097fe80cd94d87985f4861c37e675c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:06:51 UTC) from {username='kudu', principal='kudu/127.31.131.126@KRBTEST.COM'} at 127.0.0.1:47746
I20240507 02:06:51.887120 1666 tablet_service.cc:1514] Processing DeleteTablet for tablet 75097fe80cd94d87985f4861c37e675c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:06:51 UTC) from {username='kudu', principal='kudu/127.31.131.126@KRBTEST.COM'} at 127.0.0.1:51460
I20240507 02:06:51.887192 1808 tablet_service.cc:1514] Processing DeleteTablet for tablet 75097fe80cd94d87985f4861c37e675c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:06:51 UTC) from {username='kudu', principal='kudu/127.31.131.126@KRBTEST.COM'} at 127.0.0.1:43084
I20240507 02:06:51.887209 2079 tablet_replica.cc:330] T 75097fe80cd94d87985f4861c37e675c P bc276328e6f24b5aa91552aa9fc95522: stopping tablet replica
I20240507 02:06:51.887319 2079 raft_consensus.cc:2229] T 75097fe80cd94d87985f4861c37e675c P bc276328e6f24b5aa91552aa9fc95522 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 02:06:51.887373 2079 raft_consensus.cc:2258] T 75097fe80cd94d87985f4861c37e675c P bc276328e6f24b5aa91552aa9fc95522 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 02:06:51.887405 1529 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59114:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 02:06:51.887560 1529 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 02:06:51.887745 2081 tablet_replica.cc:330] T 75097fe80cd94d87985f4861c37e675c P af65885f5957407c938f4af9bc4f7bdc: stopping tablet replica
I20240507 02:06:51.887751 2079 ts_tablet_manager.cc:1875] T 75097fe80cd94d87985f4861c37e675c P bc276328e6f24b5aa91552aa9fc95522: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 02:06:51.887821 2081 raft_consensus.cc:2229] T 75097fe80cd94d87985f4861c37e675c P af65885f5957407c938f4af9bc4f7bdc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 02:06:51.887863 2081 raft_consensus.cc:2258] T 75097fe80cd94d87985f4861c37e675c P af65885f5957407c938f4af9bc4f7bdc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 02:06:51.888662 1948 tablet_service.cc:1467] Processing CreateTablet for tablet 43bf1449cbe04d8e92d66cfbacf1a500 (DEFAULT_TABLE table=test-table [id=c0ed834dcb644b8d961e4ee75e9240ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:06:51.888657 1808 tablet_service.cc:1467] Processing CreateTablet for tablet 43bf1449cbe04d8e92d66cfbacf1a500 (DEFAULT_TABLE table=test-table [id=c0ed834dcb644b8d961e4ee75e9240ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:06:51.888761 1948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43bf1449cbe04d8e92d66cfbacf1a500. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:06:51.888761 1808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43bf1449cbe04d8e92d66cfbacf1a500. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:06:51.889366 2079 ts_tablet_manager.cc:1888] T 75097fe80cd94d87985f4861c37e675c P bc276328e6f24b5aa91552aa9fc95522: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240507 02:06:51.889421 2079 log.cc:1196] T 75097fe80cd94d87985f4861c37e675c P bc276328e6f24b5aa91552aa9fc95522: Deleting WAL directory at /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/ts-2/wal/wals/75097fe80cd94d87985f4861c37e675c
I20240507 02:06:51.889672 2079 ts_tablet_manager.cc:1909] T 75097fe80cd94d87985f4861c37e675c P bc276328e6f24b5aa91552aa9fc95522: Deleting consensus metadata
I20240507 02:06:51.889925 2036 tablet_bootstrap.cc:492] T 43bf1449cbe04d8e92d66cfbacf1a500 P af65885f5957407c938f4af9bc4f7bdc: Bootstrap starting.
I20240507 02:06:51.889925 2040 tablet_bootstrap.cc:492] T 43bf1449cbe04d8e92d66cfbacf1a500 P bc276328e6f24b5aa91552aa9fc95522: Bootstrap starting.
I20240507 02:06:51.890239 2040 tablet_bootstrap.cc:654] T 43bf1449cbe04d8e92d66cfbacf1a500 P bc276328e6f24b5aa91552aa9fc95522: Neither blocks nor log segments found. Creating new log.
I20240507 02:06:51.890239 2036 tablet_bootstrap.cc:654] T 43bf1449cbe04d8e92d66cfbacf1a500 P af65885f5957407c938f4af9bc4f7bdc: Neither blocks nor log segments found. Creating new log.
I20240507 02:06:51.890758 2080 tablet_replica.cc:330] T 75097fe80cd94d87985f4861c37e675c P eb01f5a060104233b45cb74c075401da: stopping tablet replica
I20240507 02:06:51.890792 1516 catalog_manager.cc:4830] TS bc276328e6f24b5aa91552aa9fc95522 (127.31.131.67:36899): tablet 75097fe80cd94d87985f4861c37e675c (table test-table [id=1887317c07a44f3cbb223d7220b66d34]) successfully deleted
I20240507 02:06:51.890834 2080 raft_consensus.cc:2229] T 75097fe80cd94d87985f4861c37e675c P eb01f5a060104233b45cb74c075401da [term 1 LEADER]: Raft consensus shutting down.
I20240507 02:06:51.890913 2080 raft_consensus.cc:2258] T 75097fe80cd94d87985f4861c37e675c P eb01f5a060104233b45cb74c075401da [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 02:06:51.890995 2040 tablet_bootstrap.cc:492] T 43bf1449cbe04d8e92d66cfbacf1a500 P bc276328e6f24b5aa91552aa9fc95522: No bootstrap required, opened a new log
I20240507 02:06:51.891041 2040 ts_tablet_manager.cc:1367] T 43bf1449cbe04d8e92d66cfbacf1a500 P bc276328e6f24b5aa91552aa9fc95522: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240507 02:06:51.891165 2040 raft_consensus.cc:348] T 43bf1449cbe04d8e92d66cfbacf1a500 P bc276328e6f24b5aa91552aa9fc95522 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb01f5a060104233b45cb74c075401da" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33363 } } peers { permanent_uuid: "af65885f5957407c938f4af9bc4f7bdc" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 43071 } } peers { permanent_uuid: "bc276328e6f24b5aa91552aa9fc95522" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36899 } }
I20240507 02:06:51.891237 2040 raft_consensus.cc:374] T 43bf1449cbe04d8e92d66cfbacf1a500 P bc276328e6f24b5aa91552aa9fc95522 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:06:51.891218 1666 tablet_service.cc:1467] Processing CreateTablet for tablet 43bf1449cbe04d8e92d66cfbacf1a500 (DEFAULT_TABLE table=test-table [id=c0ed834dcb644b8d961e4ee75e9240ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:06:51.891255 2040 raft_consensus.cc:729] T 43bf1449cbe04d8e92d66cfbacf1a500 P bc276328e6f24b5aa91552aa9fc95522 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc276328e6f24b5aa91552aa9fc95522, State: Initialized, Role: FOLLOWER
I20240507 02:06:51.891274 2080 ts_tablet_manager.cc:1875] T 75097fe80cd94d87985f4861c37e675c P eb01f5a060104233b45cb74c075401da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 02:06:51.891292 2040 consensus_queue.cc:260] T 43bf1449cbe04d8e92d66cfbacf1a500 P bc276328e6f24b5aa91552aa9fc95522 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb01f5a060104233b45cb74c075401da" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33363 } } peers { permanent_uuid: "af65885f5957407c938f4af9bc4f7bdc" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 43071 } } peers { permanent_uuid: "bc276328e6f24b5aa91552aa9fc95522" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36899 } }
I20240507 02:06:51.891328 1666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43bf1449cbe04d8e92d66cfbacf1a500. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:06:51.891364 2040 ts_tablet_manager.cc:1398] T 43bf1449cbe04d8e92d66cfbacf1a500 P bc276328e6f24b5aa91552aa9fc95522: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:51.891716 2036 tablet_bootstrap.cc:492] T 43bf1449cbe04d8e92d66cfbacf1a500 P af65885f5957407c938f4af9bc4f7bdc: No bootstrap required, opened a new log
I20240507 02:06:51.891753 2036 ts_tablet_manager.cc:1367] T 43bf1449cbe04d8e92d66cfbacf1a500 P af65885f5957407c938f4af9bc4f7bdc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240507 02:06:51.891876 2036 raft_consensus.cc:348] T 43bf1449cbe04d8e92d66cfbacf1a500 P af65885f5957407c938f4af9bc4f7bdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb01f5a060104233b45cb74c075401da" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33363 } } peers { permanent_uuid: "af65885f5957407c938f4af9bc4f7bdc" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 43071 } } peers { permanent_uuid: "bc276328e6f24b5aa91552aa9fc95522" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36899 } }
I20240507 02:06:51.891927 2036 raft_consensus.cc:374] T 43bf1449cbe04d8e92d66cfbacf1a500 P af65885f5957407c938f4af9bc4f7bdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:06:51.891945 2036 raft_consensus.cc:729] T 43bf1449cbe04d8e92d66cfbacf1a500 P af65885f5957407c938f4af9bc4f7bdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af65885f5957407c938f4af9bc4f7bdc, State: Initialized, Role: FOLLOWER
I20240507 02:06:51.891992 2036 consensus_queue.cc:260] T 43bf1449cbe04d8e92d66cfbacf1a500 P af65885f5957407c938f4af9bc4f7bdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb01f5a060104233b45cb74c075401da" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33363 } } peers { permanent_uuid: "af65885f5957407c938f4af9bc4f7bdc" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 43071 } } peers { permanent_uuid: "bc276328e6f24b5aa91552aa9fc95522" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36899 } }
I20240507 02:06:51.892063 2036 ts_tablet_manager.cc:1398] T 43bf1449cbe04d8e92d66cfbacf1a500 P af65885f5957407c938f4af9bc4f7bdc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:51.892647 2035 tablet_bootstrap.cc:492] T 43bf1449cbe04d8e92d66cfbacf1a500 P eb01f5a060104233b45cb74c075401da: Bootstrap starting.
I20240507 02:06:51.892892 2080 ts_tablet_manager.cc:1888] T 75097fe80cd94d87985f4861c37e675c P eb01f5a060104233b45cb74c075401da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240507 02:06:51.892948 2080 log.cc:1196] T 75097fe80cd94d87985f4861c37e675c P eb01f5a060104233b45cb74c075401da: Deleting WAL directory at /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/ts-0/wal/wals/75097fe80cd94d87985f4861c37e675c
I20240507 02:06:51.892983 2035 tablet_bootstrap.cc:654] T 43bf1449cbe04d8e92d66cfbacf1a500 P eb01f5a060104233b45cb74c075401da: Neither blocks nor log segments found. Creating new log.
I20240507 02:06:51.893201 2080 ts_tablet_manager.cc:1909] T 75097fe80cd94d87985f4861c37e675c P eb01f5a060104233b45cb74c075401da: Deleting consensus metadata
I20240507 02:06:51.893510 1516 catalog_manager.cc:4830] TS eb01f5a060104233b45cb74c075401da (127.31.131.65:33363): tablet 75097fe80cd94d87985f4861c37e675c (table test-table [id=1887317c07a44f3cbb223d7220b66d34]) successfully deleted
I20240507 02:06:51.893595 2081 ts_tablet_manager.cc:1875] T 75097fe80cd94d87985f4861c37e675c P af65885f5957407c938f4af9bc4f7bdc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 02:06:51.893870 2035 tablet_bootstrap.cc:492] T 43bf1449cbe04d8e92d66cfbacf1a500 P eb01f5a060104233b45cb74c075401da: No bootstrap required, opened a new log
I20240507 02:06:51.893913 2035 ts_tablet_manager.cc:1367] T 43bf1449cbe04d8e92d66cfbacf1a500 P eb01f5a060104233b45cb74c075401da: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240507 02:06:51.894029 2035 raft_consensus.cc:348] T 43bf1449cbe04d8e92d66cfbacf1a500 P eb01f5a060104233b45cb74c075401da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb01f5a060104233b45cb74c075401da" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33363 } } peers { permanent_uuid: "af65885f5957407c938f4af9bc4f7bdc" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 43071 } } peers { permanent_uuid: "bc276328e6f24b5aa91552aa9fc95522" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36899 } }
I20240507 02:06:51.894078 2035 raft_consensus.cc:374] T 43bf1449cbe04d8e92d66cfbacf1a500 P eb01f5a060104233b45cb74c075401da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:06:51.894093 2035 raft_consensus.cc:729] T 43bf1449cbe04d8e92d66cfbacf1a500 P eb01f5a060104233b45cb74c075401da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb01f5a060104233b45cb74c075401da, State: Initialized, Role: FOLLOWER
I20240507 02:06:51.894129 2035 consensus_queue.cc:260] T 43bf1449cbe04d8e92d66cfbacf1a500 P eb01f5a060104233b45cb74c075401da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb01f5a060104233b45cb74c075401da" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33363 } } peers { permanent_uuid: "af65885f5957407c938f4af9bc4f7bdc" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 43071 } } peers { permanent_uuid: "bc276328e6f24b5aa91552aa9fc95522" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36899 } }
I20240507 02:06:51.894208 2035 ts_tablet_manager.cc:1398] T 43bf1449cbe04d8e92d66cfbacf1a500 P eb01f5a060104233b45cb74c075401da: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:51.895109 2081 ts_tablet_manager.cc:1888] T 75097fe80cd94d87985f4861c37e675c P af65885f5957407c938f4af9bc4f7bdc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240507 02:06:51.895179 2081 log.cc:1196] T 75097fe80cd94d87985f4861c37e675c P af65885f5957407c938f4af9bc4f7bdc: Deleting WAL directory at /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/ts-1/wal/wals/75097fe80cd94d87985f4861c37e675c
I20240507 02:06:51.895416 2081 ts_tablet_manager.cc:1909] T 75097fe80cd94d87985f4861c37e675c P af65885f5957407c938f4af9bc4f7bdc: Deleting consensus metadata
I20240507 02:06:51.895714 1513 catalog_manager.cc:4830] TS af65885f5957407c938f4af9bc4f7bdc (127.31.131.66:43071): tablet 75097fe80cd94d87985f4861c37e675c (table test-table [id=1887317c07a44f3cbb223d7220b66d34]) successfully deleted
I20240507 02:06:52.033739 2068 raft_consensus.cc:482] T 43bf1449cbe04d8e92d66cfbacf1a500 P eb01f5a060104233b45cb74c075401da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 02:06:52.033833 2068 raft_consensus.cc:504] T 43bf1449cbe04d8e92d66cfbacf1a500 P eb01f5a060104233b45cb74c075401da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb01f5a060104233b45cb74c075401da" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33363 } } peers { permanent_uuid: "af65885f5957407c938f4af9bc4f7bdc" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 43071 } } peers { permanent_uuid: "bc276328e6f24b5aa91552aa9fc95522" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36899 } }
I20240507 02:06:52.034013 2068 leader_election.cc:288] T 43bf1449cbe04d8e92d66cfbacf1a500 P eb01f5a060104233b45cb74c075401da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af65885f5957407c938f4af9bc4f7bdc (127.31.131.66:43071), bc276328e6f24b5aa91552aa9fc95522 (127.31.131.67:36899)
I20240507 02:06:52.034217 1827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43bf1449cbe04d8e92d66cfbacf1a500" candidate_uuid: "eb01f5a060104233b45cb74c075401da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af65885f5957407c938f4af9bc4f7bdc" is_pre_election: true
I20240507 02:06:52.034232 1968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43bf1449cbe04d8e92d66cfbacf1a500" candidate_uuid: "eb01f5a060104233b45cb74c075401da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc276328e6f24b5aa91552aa9fc95522" is_pre_election: true
I20240507 02:06:52.034310 1827 raft_consensus.cc:2454] T 43bf1449cbe04d8e92d66cfbacf1a500 P af65885f5957407c938f4af9bc4f7bdc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb01f5a060104233b45cb74c075401da in term 0.
I20240507 02:06:52.034334 1968 raft_consensus.cc:2454] T 43bf1449cbe04d8e92d66cfbacf1a500 P bc276328e6f24b5aa91552aa9fc95522 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb01f5a060104233b45cb74c075401da in term 0.
I20240507 02:06:52.034504 1621 leader_election.cc:302] T 43bf1449cbe04d8e92d66cfbacf1a500 P eb01f5a060104233b45cb74c075401da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc276328e6f24b5aa91552aa9fc95522, eb01f5a060104233b45cb74c075401da; no voters:
I20240507 02:06:52.034628 2068 raft_consensus.cc:2781] T 43bf1449cbe04d8e92d66cfbacf1a500 P eb01f5a060104233b45cb74c075401da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 02:06:52.034682 2068 raft_consensus.cc:482] T 43bf1449cbe04d8e92d66cfbacf1a500 P eb01f5a060104233b45cb74c075401da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 02:06:52.034703 2068 raft_consensus.cc:3037] T 43bf1449cbe04d8e92d66cfbacf1a500 P eb01f5a060104233b45cb74c075401da [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:06:52.035383 2068 raft_consensus.cc:504] T 43bf1449cbe04d8e92d66cfbacf1a500 P eb01f5a060104233b45cb74c075401da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb01f5a060104233b45cb74c075401da" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33363 } } peers { permanent_uuid: "af65885f5957407c938f4af9bc4f7bdc" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 43071 } } peers { permanent_uuid: "bc276328e6f24b5aa91552aa9fc95522" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36899 } }
I20240507 02:06:52.035509 2068 leader_election.cc:288] T 43bf1449cbe04d8e92d66cfbacf1a500 P eb01f5a060104233b45cb74c075401da [CANDIDATE]: Term 1 election: Requested vote from peers af65885f5957407c938f4af9bc4f7bdc (127.31.131.66:43071), bc276328e6f24b5aa91552aa9fc95522 (127.31.131.67:36899)
I20240507 02:06:52.035640 1968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43bf1449cbe04d8e92d66cfbacf1a500" candidate_uuid: "eb01f5a060104233b45cb74c075401da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc276328e6f24b5aa91552aa9fc95522"
I20240507 02:06:52.035699 1968 raft_consensus.cc:3037] T 43bf1449cbe04d8e92d66cfbacf1a500 P bc276328e6f24b5aa91552aa9fc95522 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:06:52.035661 1827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43bf1449cbe04d8e92d66cfbacf1a500" candidate_uuid: "eb01f5a060104233b45cb74c075401da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af65885f5957407c938f4af9bc4f7bdc"
I20240507 02:06:52.035733 1827 raft_consensus.cc:3037] T 43bf1449cbe04d8e92d66cfbacf1a500 P af65885f5957407c938f4af9bc4f7bdc [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:06:52.036398 1968 raft_consensus.cc:2454] T 43bf1449cbe04d8e92d66cfbacf1a500 P bc276328e6f24b5aa91552aa9fc95522 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb01f5a060104233b45cb74c075401da in term 1.
I20240507 02:06:52.036541 1621 leader_election.cc:302] T 43bf1449cbe04d8e92d66cfbacf1a500 P eb01f5a060104233b45cb74c075401da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc276328e6f24b5aa91552aa9fc95522, eb01f5a060104233b45cb74c075401da; no voters:
I20240507 02:06:52.036548 1827 raft_consensus.cc:2454] T 43bf1449cbe04d8e92d66cfbacf1a500 P af65885f5957407c938f4af9bc4f7bdc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb01f5a060104233b45cb74c075401da in term 1.
I20240507 02:06:52.036613 2068 raft_consensus.cc:2781] T 43bf1449cbe04d8e92d66cfbacf1a500 P eb01f5a060104233b45cb74c075401da [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:06:52.036664 2068 raft_consensus.cc:686] T 43bf1449cbe04d8e92d66cfbacf1a500 P eb01f5a060104233b45cb74c075401da [term 1 LEADER]: Becoming Leader. State: Replica: eb01f5a060104233b45cb74c075401da, State: Running, Role: LEADER
I20240507 02:06:52.036715 2068 consensus_queue.cc:237] T 43bf1449cbe04d8e92d66cfbacf1a500 P eb01f5a060104233b45cb74c075401da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb01f5a060104233b45cb74c075401da" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33363 } } peers { permanent_uuid: "af65885f5957407c938f4af9bc4f7bdc" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 43071 } } peers { permanent_uuid: "bc276328e6f24b5aa91552aa9fc95522" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36899 } }
I20240507 02:06:52.037276 1529 catalog_manager.cc:5484] T 43bf1449cbe04d8e92d66cfbacf1a500 P eb01f5a060104233b45cb74c075401da reported cstate change: term changed from 0 to 1, leader changed from <none> to eb01f5a060104233b45cb74c075401da (127.31.131.65). New cstate: current_term: 1 leader_uuid: "eb01f5a060104233b45cb74c075401da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb01f5a060104233b45cb74c075401da" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33363 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af65885f5957407c938f4af9bc4f7bdc" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 43071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc276328e6f24b5aa91552aa9fc95522" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36899 } health_report { overall_health: UNKNOWN } } }
I20240507 02:06:52.074007 1827 raft_consensus.cc:1261] T 43bf1449cbe04d8e92d66cfbacf1a500 P af65885f5957407c938f4af9bc4f7bdc [term 1 FOLLOWER]: Refusing update from remote peer eb01f5a060104233b45cb74c075401da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 02:06:52.074049 1968 raft_consensus.cc:1261] T 43bf1449cbe04d8e92d66cfbacf1a500 P bc276328e6f24b5aa91552aa9fc95522 [term 1 FOLLOWER]: Refusing update from remote peer eb01f5a060104233b45cb74c075401da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 02:06:52.074203 2068 consensus_queue.cc:1035] T 43bf1449cbe04d8e92d66cfbacf1a500 P eb01f5a060104233b45cb74c075401da [LEADER]: Connected to new peer: Peer: permanent_uuid: "af65885f5957407c938f4af9bc4f7bdc" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 43071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 02:06:52.074327 2074 consensus_queue.cc:1035] T 43bf1449cbe04d8e92d66cfbacf1a500 P eb01f5a060104233b45cb74c075401da [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc276328e6f24b5aa91552aa9fc95522" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240507 02:06:52.078759 2061 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20240507 02:06:53.596279 1529 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59114:
table { table_name: "test-table" } modify_external_catalogs: true
I20240507 02:06:53.596386 1529 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59114:
table { table_name: "test-table" } modify_external_catalogs: true
I20240507 02:06:53.597498 1529 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 27251517e7da4be8bebf6abfb08c2a57: Sending DeleteTablet for 3 replicas of tablet 43bf1449cbe04d8e92d66cfbacf1a500
I20240507 02:06:53.597872 1666 tablet_service.cc:1514] Processing DeleteTablet for tablet 43bf1449cbe04d8e92d66cfbacf1a500 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:06:53 UTC) from {username='kudu', principal='kudu/127.31.131.126@KRBTEST.COM'} at 127.0.0.1:51460
I20240507 02:06:53.598060 1808 tablet_service.cc:1514] Processing DeleteTablet for tablet 43bf1449cbe04d8e92d66cfbacf1a500 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:06:53 UTC) from {username='kudu', principal='kudu/127.31.131.126@KRBTEST.COM'} at 127.0.0.1:43084
I20240507 02:06:53.598234 2110 tablet_replica.cc:330] T 43bf1449cbe04d8e92d66cfbacf1a500 P af65885f5957407c938f4af9bc4f7bdc: stopping tablet replica
I20240507 02:06:53.598304 2110 raft_consensus.cc:2229] T 43bf1449cbe04d8e92d66cfbacf1a500 P af65885f5957407c938f4af9bc4f7bdc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 02:06:53.598338 2110 raft_consensus.cc:2258] T 43bf1449cbe04d8e92d66cfbacf1a500 P af65885f5957407c938f4af9bc4f7bdc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 02:06:53.598475 1948 tablet_service.cc:1514] Processing DeleteTablet for tablet 43bf1449cbe04d8e92d66cfbacf1a500 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:06:53 UTC) from {username='kudu', principal='kudu/127.31.131.126@KRBTEST.COM'} at 127.0.0.1:47746
I20240507 02:06:53.598628 2110 ts_tablet_manager.cc:1875] T 43bf1449cbe04d8e92d66cfbacf1a500 P af65885f5957407c938f4af9bc4f7bdc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 02:06:53.598688 2111 tablet_replica.cc:330] T 43bf1449cbe04d8e92d66cfbacf1a500 P bc276328e6f24b5aa91552aa9fc95522: stopping tablet replica
I20240507 02:06:53.598729 2111 raft_consensus.cc:2229] T 43bf1449cbe04d8e92d66cfbacf1a500 P bc276328e6f24b5aa91552aa9fc95522 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 02:06:53.598757 2111 raft_consensus.cc:2258] T 43bf1449cbe04d8e92d66cfbacf1a500 P bc276328e6f24b5aa91552aa9fc95522 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 02:06:53.598971 2111 ts_tablet_manager.cc:1875] T 43bf1449cbe04d8e92d66cfbacf1a500 P bc276328e6f24b5aa91552aa9fc95522: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 02:06:53.599946 2109 tablet_replica.cc:330] T 43bf1449cbe04d8e92d66cfbacf1a500 P eb01f5a060104233b45cb74c075401da: stopping tablet replica
I20240507 02:06:53.600060 2109 raft_consensus.cc:2229] T 43bf1449cbe04d8e92d66cfbacf1a500 P eb01f5a060104233b45cb74c075401da [term 1 LEADER]: Raft consensus shutting down.
I20240507 02:06:53.600178 2109 raft_consensus.cc:2258] T 43bf1449cbe04d8e92d66cfbacf1a500 P eb01f5a060104233b45cb74c075401da [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 02:06:53.600209 2110 ts_tablet_manager.cc:1888] T 43bf1449cbe04d8e92d66cfbacf1a500 P af65885f5957407c938f4af9bc4f7bdc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20240507 02:06:53.600291 2110 log.cc:1196] T 43bf1449cbe04d8e92d66cfbacf1a500 P af65885f5957407c938f4af9bc4f7bdc: Deleting WAL directory at /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/ts-1/wal/wals/43bf1449cbe04d8e92d66cfbacf1a500
I20240507 02:06:53.600446 2111 ts_tablet_manager.cc:1888] T 43bf1449cbe04d8e92d66cfbacf1a500 P bc276328e6f24b5aa91552aa9fc95522: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20240507 02:06:53.600486 2109 ts_tablet_manager.cc:1875] T 43bf1449cbe04d8e92d66cfbacf1a500 P eb01f5a060104233b45cb74c075401da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 02:06:53.600548 2111 log.cc:1196] T 43bf1449cbe04d8e92d66cfbacf1a500 P bc276328e6f24b5aa91552aa9fc95522: Deleting WAL directory at /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/ts-2/wal/wals/43bf1449cbe04d8e92d66cfbacf1a500
I20240507 02:06:53.600653 2110 ts_tablet_manager.cc:1909] T 43bf1449cbe04d8e92d66cfbacf1a500 P af65885f5957407c938f4af9bc4f7bdc: Deleting consensus metadata
I20240507 02:06:53.600819 2111 ts_tablet_manager.cc:1909] T 43bf1449cbe04d8e92d66cfbacf1a500 P bc276328e6f24b5aa91552aa9fc95522: Deleting consensus metadata
I20240507 02:06:53.601013 1513 catalog_manager.cc:4830] TS af65885f5957407c938f4af9bc4f7bdc (127.31.131.66:43071): tablet 43bf1449cbe04d8e92d66cfbacf1a500 (table test-table [id=c0ed834dcb644b8d961e4ee75e9240ac]) successfully deleted
I20240507 02:06:53.601147 1516 catalog_manager.cc:4830] TS bc276328e6f24b5aa91552aa9fc95522 (127.31.131.67:36899): tablet 43bf1449cbe04d8e92d66cfbacf1a500 (table test-table [id=c0ed834dcb644b8d961e4ee75e9240ac]) successfully deleted
I20240507 02:06:53.601992 2109 ts_tablet_manager.cc:1888] T 43bf1449cbe04d8e92d66cfbacf1a500 P eb01f5a060104233b45cb74c075401da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20240507 02:06:53.602064 2109 log.cc:1196] T 43bf1449cbe04d8e92d66cfbacf1a500 P eb01f5a060104233b45cb74c075401da: Deleting WAL directory at /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/ts-0/wal/wals/43bf1449cbe04d8e92d66cfbacf1a500
I20240507 02:06:53.602306 2109 ts_tablet_manager.cc:1909] T 43bf1449cbe04d8e92d66cfbacf1a500 P eb01f5a060104233b45cb74c075401da: Deleting consensus metadata
I20240507 02:06:53.602597 1516 catalog_manager.cc:4830] TS eb01f5a060104233b45cb74c075401da (127.31.131.65:33363): tablet 43bf1449cbe04d8e92d66cfbacf1a500 (table test-table [id=c0ed834dcb644b8d961e4ee75e9240ac]) successfully deleted
May 07 02:06:53 dist-test-slave-v0bg krb5kdc[1484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047613, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
May 07 02:06:53 dist-test-slave-v0bg krb5kdc[1484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047613, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:06:53.619510 2119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:53.613137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34374 (local address 127.31.131.126:34169)
0507 02:06:53.613264 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:53.613266 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:53.613292 (+ 26us) server_negotiation.cc:407] Connection header received
0507 02:06:53.613347 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:53.613347 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:53.613359 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:53.613426 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:06:53.614006 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:53.614523 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:53.615135 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:53.615318 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:53.617111 (+ 1793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:06:53.617116 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:06:53.617116 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:06:53.617122 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:06:53.618429 (+ 1307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:53.618834 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:53.618835 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:53.618836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:53.618866 (+ 30us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:53.619016 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:53.619017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:53.619017 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:53.619105 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:06:53.619200 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:53.619310 (+ 110us) server_negotiation.cc:299] Negotiation successful
0507 02:06:53.619405 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":78,"threads_started":1}
I20240507 02:06:53.620827 1529 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34374:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 02:06:53.620966 1529 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 02:06:53.622186 1666 tablet_service.cc:1467] Processing CreateTablet for tablet 3a73da58ee9544689c3c37153bc744e8 (DEFAULT_TABLE table=test-table [id=f7f8d0cd02f64db5b6cc83251b7e6f86]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:06:53.622186 1948 tablet_service.cc:1467] Processing CreateTablet for tablet 3a73da58ee9544689c3c37153bc744e8 (DEFAULT_TABLE table=test-table [id=f7f8d0cd02f64db5b6cc83251b7e6f86]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:06:53.622205 1808 tablet_service.cc:1467] Processing CreateTablet for tablet 3a73da58ee9544689c3c37153bc744e8 (DEFAULT_TABLE table=test-table [id=f7f8d0cd02f64db5b6cc83251b7e6f86]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:06:53.622301 1808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a73da58ee9544689c3c37153bc744e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:06:53.622304 1666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a73da58ee9544689c3c37153bc744e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:06:53.622303 1948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a73da58ee9544689c3c37153bc744e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:06:53.623275 2120 tablet_bootstrap.cc:492] T 3a73da58ee9544689c3c37153bc744e8 P af65885f5957407c938f4af9bc4f7bdc: Bootstrap starting.
I20240507 02:06:53.623590 2120 tablet_bootstrap.cc:654] T 3a73da58ee9544689c3c37153bc744e8 P af65885f5957407c938f4af9bc4f7bdc: Neither blocks nor log segments found. Creating new log.
I20240507 02:06:53.623735 2122 tablet_bootstrap.cc:492] T 3a73da58ee9544689c3c37153bc744e8 P eb01f5a060104233b45cb74c075401da: Bootstrap starting.
I20240507 02:06:53.624089 2122 tablet_bootstrap.cc:654] T 3a73da58ee9544689c3c37153bc744e8 P eb01f5a060104233b45cb74c075401da: Neither blocks nor log segments found. Creating new log.
I20240507 02:06:53.624362 2121 tablet_bootstrap.cc:492] T 3a73da58ee9544689c3c37153bc744e8 P bc276328e6f24b5aa91552aa9fc95522: Bootstrap starting.
I20240507 02:06:53.624454 2120 tablet_bootstrap.cc:492] T 3a73da58ee9544689c3c37153bc744e8 P af65885f5957407c938f4af9bc4f7bdc: No bootstrap required, opened a new log
I20240507 02:06:53.624502 2120 ts_tablet_manager.cc:1367] T 3a73da58ee9544689c3c37153bc744e8 P af65885f5957407c938f4af9bc4f7bdc: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240507 02:06:53.624640 2122 tablet_bootstrap.cc:492] T 3a73da58ee9544689c3c37153bc744e8 P eb01f5a060104233b45cb74c075401da: No bootstrap required, opened a new log
I20240507 02:06:53.624641 2120 raft_consensus.cc:348] T 3a73da58ee9544689c3c37153bc744e8 P af65885f5957407c938f4af9bc4f7bdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb01f5a060104233b45cb74c075401da" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33363 } } peers { permanent_uuid: "af65885f5957407c938f4af9bc4f7bdc" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 43071 } } peers { permanent_uuid: "bc276328e6f24b5aa91552aa9fc95522" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36899 } }
I20240507 02:06:53.624689 2122 ts_tablet_manager.cc:1367] T 3a73da58ee9544689c3c37153bc744e8 P eb01f5a060104233b45cb74c075401da: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240507 02:06:53.624711 2121 tablet_bootstrap.cc:654] T 3a73da58ee9544689c3c37153bc744e8 P bc276328e6f24b5aa91552aa9fc95522: Neither blocks nor log segments found. Creating new log.
I20240507 02:06:53.624761 2120 raft_consensus.cc:374] T 3a73da58ee9544689c3c37153bc744e8 P af65885f5957407c938f4af9bc4f7bdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:06:53.624796 2120 raft_consensus.cc:729] T 3a73da58ee9544689c3c37153bc744e8 P af65885f5957407c938f4af9bc4f7bdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af65885f5957407c938f4af9bc4f7bdc, State: Initialized, Role: FOLLOWER
I20240507 02:06:53.624791 2122 raft_consensus.cc:348] T 3a73da58ee9544689c3c37153bc744e8 P eb01f5a060104233b45cb74c075401da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb01f5a060104233b45cb74c075401da" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33363 } } peers { permanent_uuid: "af65885f5957407c938f4af9bc4f7bdc" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 43071 } } peers { permanent_uuid: "bc276328e6f24b5aa91552aa9fc95522" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36899 } }
I20240507 02:06:53.624874 2122 raft_consensus.cc:374] T 3a73da58ee9544689c3c37153bc744e8 P eb01f5a060104233b45cb74c075401da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:06:53.624874 2120 consensus_queue.cc:260] T 3a73da58ee9544689c3c37153bc744e8 P af65885f5957407c938f4af9bc4f7bdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb01f5a060104233b45cb74c075401da" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33363 } } peers { permanent_uuid: "af65885f5957407c938f4af9bc4f7bdc" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 43071 } } peers { permanent_uuid: "bc276328e6f24b5aa91552aa9fc95522" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36899 } }
I20240507 02:06:53.624922 2122 raft_consensus.cc:729] T 3a73da58ee9544689c3c37153bc744e8 P eb01f5a060104233b45cb74c075401da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb01f5a060104233b45cb74c075401da, State: Initialized, Role: FOLLOWER
I20240507 02:06:53.624980 2122 consensus_queue.cc:260] T 3a73da58ee9544689c3c37153bc744e8 P eb01f5a060104233b45cb74c075401da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb01f5a060104233b45cb74c075401da" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33363 } } peers { permanent_uuid: "af65885f5957407c938f4af9bc4f7bdc" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 43071 } } peers { permanent_uuid: "bc276328e6f24b5aa91552aa9fc95522" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36899 } }
I20240507 02:06:53.625059 2120 ts_tablet_manager.cc:1398] T 3a73da58ee9544689c3c37153bc744e8 P af65885f5957407c938f4af9bc4f7bdc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240507 02:06:53.625123 2122 ts_tablet_manager.cc:1398] T 3a73da58ee9544689c3c37153bc744e8 P eb01f5a060104233b45cb74c075401da: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:53.625411 2121 tablet_bootstrap.cc:492] T 3a73da58ee9544689c3c37153bc744e8 P bc276328e6f24b5aa91552aa9fc95522: No bootstrap required, opened a new log
I20240507 02:06:53.625454 2121 ts_tablet_manager.cc:1367] T 3a73da58ee9544689c3c37153bc744e8 P bc276328e6f24b5aa91552aa9fc95522: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240507 02:06:53.625797 2121 raft_consensus.cc:348] T 3a73da58ee9544689c3c37153bc744e8 P bc276328e6f24b5aa91552aa9fc95522 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb01f5a060104233b45cb74c075401da" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33363 } } peers { permanent_uuid: "af65885f5957407c938f4af9bc4f7bdc" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 43071 } } peers { permanent_uuid: "bc276328e6f24b5aa91552aa9fc95522" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36899 } }
I20240507 02:06:53.625857 2121 raft_consensus.cc:374] T 3a73da58ee9544689c3c37153bc744e8 P bc276328e6f24b5aa91552aa9fc95522 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:06:53.625869 2121 raft_consensus.cc:729] T 3a73da58ee9544689c3c37153bc744e8 P bc276328e6f24b5aa91552aa9fc95522 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc276328e6f24b5aa91552aa9fc95522, State: Initialized, Role: FOLLOWER
I20240507 02:06:53.625900 2121 consensus_queue.cc:260] T 3a73da58ee9544689c3c37153bc744e8 P bc276328e6f24b5aa91552aa9fc95522 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb01f5a060104233b45cb74c075401da" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33363 } } peers { permanent_uuid: "af65885f5957407c938f4af9bc4f7bdc" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 43071 } } peers { permanent_uuid: "bc276328e6f24b5aa91552aa9fc95522" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36899 } }
I20240507 02:06:53.626010 2121 ts_tablet_manager.cc:1398] T 3a73da58ee9544689c3c37153bc744e8 P bc276328e6f24b5aa91552aa9fc95522: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240507 02:06:53.772265 2126 raft_consensus.cc:482] T 3a73da58ee9544689c3c37153bc744e8 P af65885f5957407c938f4af9bc4f7bdc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 02:06:53.772357 2126 raft_consensus.cc:504] T 3a73da58ee9544689c3c37153bc744e8 P af65885f5957407c938f4af9bc4f7bdc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb01f5a060104233b45cb74c075401da" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33363 } } peers { permanent_uuid: "af65885f5957407c938f4af9bc4f7bdc" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 43071 } } peers { permanent_uuid: "bc276328e6f24b5aa91552aa9fc95522" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36899 } }
I20240507 02:06:53.772639 2126 leader_election.cc:288] T 3a73da58ee9544689c3c37153bc744e8 P af65885f5957407c938f4af9bc4f7bdc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb01f5a060104233b45cb74c075401da (127.31.131.65:33363), bc276328e6f24b5aa91552aa9fc95522 (127.31.131.67:36899)
I20240507 02:06:53.774971 2128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:53.772812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.65:33363 (local address 127.31.131.66:44245)
0507 02:06:53.772985 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0507 02:06:53.772997 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0507 02:06:53.773067 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:06:53.773275 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:06:53.773276 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:06:53.773279 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:06:53.773505 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:53.773507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:53.774134 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:06:53.774135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:06:53.774777 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:53.774779 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:53.774830 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:53.774835 (+ 5us) client_negotiation.cc:769] Sending connection context
0507 02:06:53.774855 (+ 20us) client_negotiation.cc:240] Negotiation successful
0507 02:06:53.774889 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":122,"threads_started":1}
I20240507 02:06:53.775091 2130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:53.773041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.67:36899 (local address 127.31.131.66:55079)
0507 02:06:53.773147 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0507 02:06:53.773156 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0507 02:06:53.773205 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:06:53.773361 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:06:53.773362 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:06:53.773364 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:06:53.773552 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:53.773553 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:53.774216 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:06:53.774217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:06:53.774815 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:53.774818 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:53.774884 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:53.774889 (+ 5us) client_negotiation.cc:769] Sending connection context
0507 02:06:53.774916 (+ 27us) client_negotiation.cc:240] Negotiation successful
0507 02:06:53.774950 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":57,"threads_started":1}
I20240507 02:06:53.775421 2129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:53.772886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.66:44245 (local address 127.31.131.65:33363)
0507 02:06:53.773034 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:53.773036 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:53.773050 (+ 14us) server_negotiation.cc:407] Connection header received
0507 02:06:53.773169 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:53.773170 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:53.773182 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:53.773251 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:06:53.773572 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:53.774062 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:53.774844 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:53.775293 (+ 449us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:53.775302 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:53.775329 (+ 27us) server_negotiation.cc:299] Negotiation successful
0507 02:06:53.775358 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":73,"threads_started":1}
I20240507 02:06:53.775470 2131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:53.773096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.66:55079 (local address 127.31.131.67:36899)
0507 02:06:53.773245 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:53.773247 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:53.773258 (+ 11us) server_negotiation.cc:407] Connection header received
0507 02:06:53.773279 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:53.773280 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:53.773290 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:53.773354 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:06:53.773626 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:53.774128 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:53.774918 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:53.775358 (+ 440us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:53.775366 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:53.775385 (+ 19us) server_negotiation.cc:299] Negotiation successful
0507 02:06:53.775408 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":98,"threads_started":1}
I20240507 02:06:53.775656 1968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a73da58ee9544689c3c37153bc744e8" candidate_uuid: "af65885f5957407c938f4af9bc4f7bdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc276328e6f24b5aa91552aa9fc95522" is_pre_election: true
I20240507 02:06:53.775734 1968 raft_consensus.cc:2454] T 3a73da58ee9544689c3c37153bc744e8 P bc276328e6f24b5aa91552aa9fc95522 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af65885f5957407c938f4af9bc4f7bdc in term 0.
I20240507 02:06:53.775732 1687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a73da58ee9544689c3c37153bc744e8" candidate_uuid: "af65885f5957407c938f4af9bc4f7bdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb01f5a060104233b45cb74c075401da" is_pre_election: true
I20240507 02:06:53.775844 1687 raft_consensus.cc:2454] T 3a73da58ee9544689c3c37153bc744e8 P eb01f5a060104233b45cb74c075401da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af65885f5957407c938f4af9bc4f7bdc in term 0.
I20240507 02:06:53.775904 1761 leader_election.cc:302] T 3a73da58ee9544689c3c37153bc744e8 P af65885f5957407c938f4af9bc4f7bdc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af65885f5957407c938f4af9bc4f7bdc, bc276328e6f24b5aa91552aa9fc95522; no voters:
I20240507 02:06:53.776041 2126 raft_consensus.cc:2781] T 3a73da58ee9544689c3c37153bc744e8 P af65885f5957407c938f4af9bc4f7bdc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 02:06:53.776103 2126 raft_consensus.cc:482] T 3a73da58ee9544689c3c37153bc744e8 P af65885f5957407c938f4af9bc4f7bdc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 02:06:53.776125 2126 raft_consensus.cc:3037] T 3a73da58ee9544689c3c37153bc744e8 P af65885f5957407c938f4af9bc4f7bdc [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:06:53.776716 2126 raft_consensus.cc:504] T 3a73da58ee9544689c3c37153bc744e8 P af65885f5957407c938f4af9bc4f7bdc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb01f5a060104233b45cb74c075401da" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33363 } } peers { permanent_uuid: "af65885f5957407c938f4af9bc4f7bdc" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 43071 } } peers { permanent_uuid: "bc276328e6f24b5aa91552aa9fc95522" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36899 } }
I20240507 02:06:53.776821 2126 leader_election.cc:288] T 3a73da58ee9544689c3c37153bc744e8 P af65885f5957407c938f4af9bc4f7bdc [CANDIDATE]: Term 1 election: Requested vote from peers eb01f5a060104233b45cb74c075401da (127.31.131.65:33363), bc276328e6f24b5aa91552aa9fc95522 (127.31.131.67:36899)
I20240507 02:06:53.776978 1687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a73da58ee9544689c3c37153bc744e8" candidate_uuid: "af65885f5957407c938f4af9bc4f7bdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb01f5a060104233b45cb74c075401da"
I20240507 02:06:53.776989 1968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a73da58ee9544689c3c37153bc744e8" candidate_uuid: "af65885f5957407c938f4af9bc4f7bdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc276328e6f24b5aa91552aa9fc95522"
I20240507 02:06:53.777033 1687 raft_consensus.cc:3037] T 3a73da58ee9544689c3c37153bc744e8 P eb01f5a060104233b45cb74c075401da [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:06:53.777046 1968 raft_consensus.cc:3037] T 3a73da58ee9544689c3c37153bc744e8 P bc276328e6f24b5aa91552aa9fc95522 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:06:53.777598 1687 raft_consensus.cc:2454] T 3a73da58ee9544689c3c37153bc744e8 P eb01f5a060104233b45cb74c075401da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af65885f5957407c938f4af9bc4f7bdc in term 1.
I20240507 02:06:53.777602 1968 raft_consensus.cc:2454] T 3a73da58ee9544689c3c37153bc744e8 P bc276328e6f24b5aa91552aa9fc95522 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af65885f5957407c938f4af9bc4f7bdc in term 1.
I20240507 02:06:53.777760 1761 leader_election.cc:302] T 3a73da58ee9544689c3c37153bc744e8 P af65885f5957407c938f4af9bc4f7bdc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af65885f5957407c938f4af9bc4f7bdc, bc276328e6f24b5aa91552aa9fc95522; no voters:
I20240507 02:06:53.777855 2126 raft_consensus.cc:2781] T 3a73da58ee9544689c3c37153bc744e8 P af65885f5957407c938f4af9bc4f7bdc [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:06:53.777994 2126 raft_consensus.cc:686] T 3a73da58ee9544689c3c37153bc744e8 P af65885f5957407c938f4af9bc4f7bdc [term 1 LEADER]: Becoming Leader. State: Replica: af65885f5957407c938f4af9bc4f7bdc, State: Running, Role: LEADER
I20240507 02:06:53.778059 2126 consensus_queue.cc:237] T 3a73da58ee9544689c3c37153bc744e8 P af65885f5957407c938f4af9bc4f7bdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb01f5a060104233b45cb74c075401da" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33363 } } peers { permanent_uuid: "af65885f5957407c938f4af9bc4f7bdc" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 43071 } } peers { permanent_uuid: "bc276328e6f24b5aa91552aa9fc95522" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36899 } }
I20240507 02:06:53.778702 1529 catalog_manager.cc:5484] T 3a73da58ee9544689c3c37153bc744e8 P af65885f5957407c938f4af9bc4f7bdc reported cstate change: term changed from 0 to 1, leader changed from <none> to af65885f5957407c938f4af9bc4f7bdc (127.31.131.66). New cstate: current_term: 1 leader_uuid: "af65885f5957407c938f4af9bc4f7bdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb01f5a060104233b45cb74c075401da" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af65885f5957407c938f4af9bc4f7bdc" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 43071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bc276328e6f24b5aa91552aa9fc95522" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36899 } health_report { overall_health: UNKNOWN } } }
I20240507 02:06:53.807098 2135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:53.805151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36688 (local address 127.31.131.66:43071)
0507 02:06:53.805285 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:53.805287 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:53.805296 (+ 9us) server_negotiation.cc:407] Connection header received
0507 02:06:53.805329 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:53.805330 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:53.805346 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:53.805418 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0507 02:06:53.805615 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:53.806109 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:53.806658 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:53.806770 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:53.806809 (+ 39us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0507 02:06:53.806901 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0507 02:06:53.806968 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:53.807015 (+ 47us) server_negotiation.cc:299] Negotiation successful
0507 02:06:53.807034 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":76,"threads_started":1}
I20240507 02:06:53.810899 2129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:53.809056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.66:46665 (local address 127.31.131.65:33363)
0507 02:06:53.809112 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:53.809114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:53.809166 (+ 52us) server_negotiation.cc:407] Connection header received
0507 02:06:53.809233 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:53.809233 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:53.809244 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:53.809288 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0507 02:06:53.809520 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:53.809956 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:53.810498 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:53.810660 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:53.810678 (+ 18us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0507 02:06:53.810707 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0507 02:06:53.810747 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:53.810808 (+ 61us) server_negotiation.cc:299] Negotiation successful
0507 02:06:53.810828 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20240507 02:06:53.810916 2138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:53.808978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.65:33363 (local address 127.31.131.66:46665)
0507 02:06:53.809100 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0507 02:06:53.809111 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0507 02:06:53.809172 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:06:53.809295 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:06:53.809296 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:06:53.809297 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0507 02:06:53.809464 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:53.809465 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:53.810024 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:06:53.810024 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:06:53.810430 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:53.810431 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:53.810508 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:53.810523 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0507 02:06:53.810759 (+ 236us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0507 02:06:53.810761 (+ 2us) client_negotiation.cc:769] Sending connection context
0507 02:06:53.810798 (+ 37us) client_negotiation.cc:240] Negotiation successful
0507 02:06:53.810836 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":54,"threads_started":1}
I20240507 02:06:53.812623 1968 raft_consensus.cc:1261] T 3a73da58ee9544689c3c37153bc744e8 P bc276328e6f24b5aa91552aa9fc95522 [term 1 FOLLOWER]: Refusing update from remote peer af65885f5957407c938f4af9bc4f7bdc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 02:06:53.812724 1687 raft_consensus.cc:1261] T 3a73da58ee9544689c3c37153bc744e8 P eb01f5a060104233b45cb74c075401da [term 1 FOLLOWER]: Refusing update from remote peer af65885f5957407c938f4af9bc4f7bdc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 02:06:53.812830 2126 consensus_queue.cc:1035] T 3a73da58ee9544689c3c37153bc744e8 P af65885f5957407c938f4af9bc4f7bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc276328e6f24b5aa91552aa9fc95522" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 02:06:53.812949 2132 consensus_queue.cc:1035] T 3a73da58ee9544689c3c37153bc744e8 P af65885f5957407c938f4af9bc4f7bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb01f5a060104233b45cb74c075401da" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 02:06:53.818979 2143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:53.817124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.66:43071 (local address 127.31.131.65:38719)
0507 02:06:53.817241 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0507 02:06:53.817249 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0507 02:06:53.817304 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:06:53.817452 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:06:53.817453 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:06:53.817454 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0507 02:06:53.817601 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:53.817602 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:53.818147 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:06:53.818147 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:06:53.818538 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:53.818540 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:53.818620 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:53.818624 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0507 02:06:53.818843 (+ 219us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0507 02:06:53.818845 (+ 2us) client_negotiation.cc:769] Sending connection context
0507 02:06:53.818891 (+ 46us) client_negotiation.cc:240] Negotiation successful
0507 02:06:53.818928 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":76,"threads_started":1}
I20240507 02:06:53.818989 2135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:53.817171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:38719 (local address 127.31.131.66:43071)
0507 02:06:53.817225 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:53.817227 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:53.817292 (+ 65us) server_negotiation.cc:407] Connection header received
0507 02:06:53.817379 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:53.817380 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:53.817391 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:53.817447 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0507 02:06:53.817641 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:53.818082 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:53.818625 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:53.818736 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:53.818753 (+ 17us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0507 02:06:53.818778 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0507 02:06:53.818832 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:53.818907 (+ 75us) server_negotiation.cc:299] Negotiation successful
0507 02:06:53.818929 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20240507 02:06:53.825628 1528 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34374:
table { table_name: "test-table" } modify_external_catalogs: true
I20240507 02:06:53.825711 1528 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34374:
table { table_name: "test-table" } modify_external_catalogs: true
I20240507 02:06:53.826251 1528 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 27251517e7da4be8bebf6abfb08c2a57: Sending DeleteTablet for 3 replicas of tablet 3a73da58ee9544689c3c37153bc744e8
I20240507 02:06:53.826450 1666 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a73da58ee9544689c3c37153bc744e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:06:53 UTC) from {username='kudu', principal='kudu/127.31.131.126@KRBTEST.COM'} at 127.0.0.1:51460
I20240507 02:06:53.826539 2109 tablet_replica.cc:330] T 3a73da58ee9544689c3c37153bc744e8 P eb01f5a060104233b45cb74c075401da: stopping tablet replica
I20240507 02:06:53.826601 2109 raft_consensus.cc:2229] T 3a73da58ee9544689c3c37153bc744e8 P eb01f5a060104233b45cb74c075401da [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 02:06:53.826632 2109 raft_consensus.cc:2258] T 3a73da58ee9544689c3c37153bc744e8 P eb01f5a060104233b45cb74c075401da [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 02:06:53.826756 1528 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34374:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 02:06:53.826880 1528 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 02:06:53.826916 1948 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a73da58ee9544689c3c37153bc744e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:06:53 UTC) from {username='kudu', principal='kudu/127.31.131.126@KRBTEST.COM'} at 127.0.0.1:47746
I20240507 02:06:53.826920 2109 ts_tablet_manager.cc:1875] T 3a73da58ee9544689c3c37153bc744e8 P eb01f5a060104233b45cb74c075401da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 02:06:53.826988 2111 tablet_replica.cc:330] T 3a73da58ee9544689c3c37153bc744e8 P bc276328e6f24b5aa91552aa9fc95522: stopping tablet replica
I20240507 02:06:53.827028 2111 raft_consensus.cc:2229] T 3a73da58ee9544689c3c37153bc744e8 P bc276328e6f24b5aa91552aa9fc95522 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 02:06:53.827056 2111 raft_consensus.cc:2258] T 3a73da58ee9544689c3c37153bc744e8 P bc276328e6f24b5aa91552aa9fc95522 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 02:06:53.827317 1808 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a73da58ee9544689c3c37153bc744e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:06:53 UTC) from {username='kudu', principal='kudu/127.31.131.126@KRBTEST.COM'} at 127.0.0.1:43084
I20240507 02:06:53.827392 2110 tablet_replica.cc:330] T 3a73da58ee9544689c3c37153bc744e8 P af65885f5957407c938f4af9bc4f7bdc: stopping tablet replica
I20240507 02:06:53.827428 2110 raft_consensus.cc:2229] T 3a73da58ee9544689c3c37153bc744e8 P af65885f5957407c938f4af9bc4f7bdc [term 1 LEADER]: Raft consensus shutting down.
I20240507 02:06:53.827497 2110 raft_consensus.cc:2258] T 3a73da58ee9544689c3c37153bc744e8 P af65885f5957407c938f4af9bc4f7bdc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 02:06:53.827803 2110 ts_tablet_manager.cc:1875] T 3a73da58ee9544689c3c37153bc744e8 P af65885f5957407c938f4af9bc4f7bdc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 02:06:53.828485 2109 ts_tablet_manager.cc:1888] T 3a73da58ee9544689c3c37153bc744e8 P eb01f5a060104233b45cb74c075401da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240507 02:06:53.828533 2109 log.cc:1196] T 3a73da58ee9544689c3c37153bc744e8 P eb01f5a060104233b45cb74c075401da: Deleting WAL directory at /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/ts-0/wal/wals/3a73da58ee9544689c3c37153bc744e8
I20240507 02:06:53.828728 2109 ts_tablet_manager.cc:1909] T 3a73da58ee9544689c3c37153bc744e8 P eb01f5a060104233b45cb74c075401da: Deleting consensus metadata
I20240507 02:06:53.829041 1516 catalog_manager.cc:4830] TS eb01f5a060104233b45cb74c075401da (127.31.131.65:33363): tablet 3a73da58ee9544689c3c37153bc744e8 (table test-table [id=f7f8d0cd02f64db5b6cc83251b7e6f86]) successfully deleted
I20240507 02:06:53.829104 1666 tablet_service.cc:1467] Processing CreateTablet for tablet 415df9d71a1b4788a99362430ddfc8e5 (DEFAULT_TABLE table=test-table [id=10b246ad68aa47d6a30d4be4e4706ab4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:06:53.829206 1666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 415df9d71a1b4788a99362430ddfc8e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:06:53.829207 1948 tablet_service.cc:1467] Processing CreateTablet for tablet 415df9d71a1b4788a99362430ddfc8e5 (DEFAULT_TABLE table=test-table [id=10b246ad68aa47d6a30d4be4e4706ab4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:06:53.829305 1948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 415df9d71a1b4788a99362430ddfc8e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:06:53.829349 1808 tablet_service.cc:1467] Processing CreateTablet for tablet 415df9d71a1b4788a99362430ddfc8e5 (DEFAULT_TABLE table=test-table [id=10b246ad68aa47d6a30d4be4e4706ab4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:06:53.829558 2111 ts_tablet_manager.cc:1875] T 3a73da58ee9544689c3c37153bc744e8 P bc276328e6f24b5aa91552aa9fc95522: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 02:06:53.830325 2122 tablet_bootstrap.cc:492] T 415df9d71a1b4788a99362430ddfc8e5 P eb01f5a060104233b45cb74c075401da: Bootstrap starting.
I20240507 02:06:53.830477 2121 tablet_bootstrap.cc:492] T 415df9d71a1b4788a99362430ddfc8e5 P bc276328e6f24b5aa91552aa9fc95522: Bootstrap starting.
I20240507 02:06:53.830654 2122 tablet_bootstrap.cc:654] T 415df9d71a1b4788a99362430ddfc8e5 P eb01f5a060104233b45cb74c075401da: Neither blocks nor log segments found. Creating new log.
I20240507 02:06:53.830857 2111 ts_tablet_manager.cc:1888] T 3a73da58ee9544689c3c37153bc744e8 P bc276328e6f24b5aa91552aa9fc95522: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240507 02:06:53.830914 2111 log.cc:1196] T 3a73da58ee9544689c3c37153bc744e8 P bc276328e6f24b5aa91552aa9fc95522: Deleting WAL directory at /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/ts-2/wal/wals/3a73da58ee9544689c3c37153bc744e8
I20240507 02:06:53.831099 2111 ts_tablet_manager.cc:1909] T 3a73da58ee9544689c3c37153bc744e8 P bc276328e6f24b5aa91552aa9fc95522: Deleting consensus metadata
I20240507 02:06:53.831141 2122 tablet_bootstrap.cc:492] T 415df9d71a1b4788a99362430ddfc8e5 P eb01f5a060104233b45cb74c075401da: No bootstrap required, opened a new log
I20240507 02:06:53.831172 2122 ts_tablet_manager.cc:1367] T 415df9d71a1b4788a99362430ddfc8e5 P eb01f5a060104233b45cb74c075401da: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20240507 02:06:53.830864 2121 tablet_bootstrap.cc:654] T 415df9d71a1b4788a99362430ddfc8e5 P bc276328e6f24b5aa91552aa9fc95522: Neither blocks nor log segments found. Creating new log.
I20240507 02:06:53.831265 2122 raft_consensus.cc:348] T 415df9d71a1b4788a99362430ddfc8e5 P eb01f5a060104233b45cb74c075401da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb01f5a060104233b45cb74c075401da" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33363 } } peers { permanent_uuid: "af65885f5957407c938f4af9bc4f7bdc" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 43071 } } peers { permanent_uuid: "bc276328e6f24b5aa91552aa9fc95522" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36899 } }
I20240507 02:06:53.831317 2122 raft_consensus.cc:374] T 415df9d71a1b4788a99362430ddfc8e5 P eb01f5a060104233b45cb74c075401da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:06:53.831333 2122 raft_consensus.cc:729] T 415df9d71a1b4788a99362430ddfc8e5 P eb01f5a060104233b45cb74c075401da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb01f5a060104233b45cb74c075401da, State: Initialized, Role: FOLLOWER
I20240507 02:06:53.831375 2122 consensus_queue.cc:260] T 415df9d71a1b4788a99362430ddfc8e5 P eb01f5a060104233b45cb74c075401da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb01f5a060104233b45cb74c075401da" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33363 } } peers { permanent_uuid: "af65885f5957407c938f4af9bc4f7bdc" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 43071 } } peers { permanent_uuid: "bc276328e6f24b5aa91552aa9fc95522" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36899 } }
I20240507 02:06:53.831457 2122 ts_tablet_manager.cc:1398] T 415df9d71a1b4788a99362430ddfc8e5 P eb01f5a060104233b45cb74c075401da: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:53.831447 1516 catalog_manager.cc:4830] TS bc276328e6f24b5aa91552aa9fc95522 (127.31.131.67:36899): tablet 3a73da58ee9544689c3c37153bc744e8 (table test-table [id=f7f8d0cd02f64db5b6cc83251b7e6f86]) successfully deleted
I20240507 02:06:53.831753 2121 tablet_bootstrap.cc:492] T 415df9d71a1b4788a99362430ddfc8e5 P bc276328e6f24b5aa91552aa9fc95522: No bootstrap required, opened a new log
I20240507 02:06:53.831776 2110 ts_tablet_manager.cc:1888] T 3a73da58ee9544689c3c37153bc744e8 P af65885f5957407c938f4af9bc4f7bdc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240507 02:06:53.831796 2121 ts_tablet_manager.cc:1367] T 415df9d71a1b4788a99362430ddfc8e5 P bc276328e6f24b5aa91552aa9fc95522: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240507 02:06:53.831786 1808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 415df9d71a1b4788a99362430ddfc8e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:06:53.831830 2110 log.cc:1196] T 3a73da58ee9544689c3c37153bc744e8 P af65885f5957407c938f4af9bc4f7bdc: Deleting WAL directory at /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/ts-1/wal/wals/3a73da58ee9544689c3c37153bc744e8
I20240507 02:06:53.831931 2121 raft_consensus.cc:348] T 415df9d71a1b4788a99362430ddfc8e5 P bc276328e6f24b5aa91552aa9fc95522 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb01f5a060104233b45cb74c075401da" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33363 } } peers { permanent_uuid: "af65885f5957407c938f4af9bc4f7bdc" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 43071 } } peers { permanent_uuid: "bc276328e6f24b5aa91552aa9fc95522" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36899 } }
I20240507 02:06:53.831990 2121 raft_consensus.cc:374] T 415df9d71a1b4788a99362430ddfc8e5 P bc276328e6f24b5aa91552aa9fc95522 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:06:53.832008 2121 raft_consensus.cc:729] T 415df9d71a1b4788a99362430ddfc8e5 P bc276328e6f24b5aa91552aa9fc95522 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc276328e6f24b5aa91552aa9fc95522, State: Initialized, Role: FOLLOWER
I20240507 02:06:53.832016 2110 ts_tablet_manager.cc:1909] T 3a73da58ee9544689c3c37153bc744e8 P af65885f5957407c938f4af9bc4f7bdc: Deleting consensus metadata
I20240507 02:06:53.832054 2121 consensus_queue.cc:260] T 415df9d71a1b4788a99362430ddfc8e5 P bc276328e6f24b5aa91552aa9fc95522 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb01f5a060104233b45cb74c075401da" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33363 } } peers { permanent_uuid: "af65885f5957407c938f4af9bc4f7bdc" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 43071 } } peers { permanent_uuid: "bc276328e6f24b5aa91552aa9fc95522" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36899 } }
I20240507 02:06:53.832136 2121 ts_tablet_manager.cc:1398] T 415df9d71a1b4788a99362430ddfc8e5 P bc276328e6f24b5aa91552aa9fc95522: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:53.832402 1513 catalog_manager.cc:4830] TS af65885f5957407c938f4af9bc4f7bdc (127.31.131.66:43071): tablet 3a73da58ee9544689c3c37153bc744e8 (table test-table [id=f7f8d0cd02f64db5b6cc83251b7e6f86]) successfully deleted
I20240507 02:06:53.832975 2120 tablet_bootstrap.cc:492] T 415df9d71a1b4788a99362430ddfc8e5 P af65885f5957407c938f4af9bc4f7bdc: Bootstrap starting.
I20240507 02:06:53.833307 2120 tablet_bootstrap.cc:654] T 415df9d71a1b4788a99362430ddfc8e5 P af65885f5957407c938f4af9bc4f7bdc: Neither blocks nor log segments found. Creating new log.
I20240507 02:06:53.833842 2120 tablet_bootstrap.cc:492] T 415df9d71a1b4788a99362430ddfc8e5 P af65885f5957407c938f4af9bc4f7bdc: No bootstrap required, opened a new log
I20240507 02:06:53.833884 2120 ts_tablet_manager.cc:1367] T 415df9d71a1b4788a99362430ddfc8e5 P af65885f5957407c938f4af9bc4f7bdc: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240507 02:06:53.834017 2120 raft_consensus.cc:348] T 415df9d71a1b4788a99362430ddfc8e5 P af65885f5957407c938f4af9bc4f7bdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb01f5a060104233b45cb74c075401da" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33363 } } peers { permanent_uuid: "af65885f5957407c938f4af9bc4f7bdc" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 43071 } } peers { permanent_uuid: "bc276328e6f24b5aa91552aa9fc95522" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36899 } }
I20240507 02:06:53.834069 2120 raft_consensus.cc:374] T 415df9d71a1b4788a99362430ddfc8e5 P af65885f5957407c938f4af9bc4f7bdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:06:53.834087 2120 raft_consensus.cc:729] T 415df9d71a1b4788a99362430ddfc8e5 P af65885f5957407c938f4af9bc4f7bdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af65885f5957407c938f4af9bc4f7bdc, State: Initialized, Role: FOLLOWER
I20240507 02:06:53.834123 2120 consensus_queue.cc:260] T 415df9d71a1b4788a99362430ddfc8e5 P af65885f5957407c938f4af9bc4f7bdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb01f5a060104233b45cb74c075401da" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33363 } } peers { permanent_uuid: "af65885f5957407c938f4af9bc4f7bdc" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 43071 } } peers { permanent_uuid: "bc276328e6f24b5aa91552aa9fc95522" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36899 } }
I20240507 02:06:53.834199 2120 ts_tablet_manager.cc:1398] T 415df9d71a1b4788a99362430ddfc8e5 P af65885f5957407c938f4af9bc4f7bdc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:54.003772 2127 raft_consensus.cc:482] T 415df9d71a1b4788a99362430ddfc8e5 P bc276328e6f24b5aa91552aa9fc95522 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 02:06:54.003873 2127 raft_consensus.cc:504] T 415df9d71a1b4788a99362430ddfc8e5 P bc276328e6f24b5aa91552aa9fc95522 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb01f5a060104233b45cb74c075401da" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33363 } } peers { permanent_uuid: "af65885f5957407c938f4af9bc4f7bdc" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 43071 } } peers { permanent_uuid: "bc276328e6f24b5aa91552aa9fc95522" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36899 } }
I20240507 02:06:54.004171 2127 leader_election.cc:288] T 415df9d71a1b4788a99362430ddfc8e5 P bc276328e6f24b5aa91552aa9fc95522 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb01f5a060104233b45cb74c075401da (127.31.131.65:33363), af65885f5957407c938f4af9bc4f7bdc (127.31.131.66:43071)
I20240507 02:06:54.006412 2150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:54.004348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.66:43071 (local address 127.31.131.67:52629)
0507 02:06:54.004515 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0507 02:06:54.004525 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0507 02:06:54.004602 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:06:54.004810 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:06:54.004811 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:06:54.004812 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:06:54.004994 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:54.004995 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:54.005577 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:06:54.005578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:06:54.006197 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:54.006199 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:54.006260 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:54.006264 (+ 4us) client_negotiation.cc:769] Sending connection context
0507 02:06:54.006292 (+ 28us) client_negotiation.cc:240] Negotiation successful
0507 02:06:54.006325 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":14,"thread_start_us":79,"threads_started":1}
I20240507 02:06:54.006417 2149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:54.004348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.65:33363 (local address 127.31.131.67:50533)
0507 02:06:54.004474 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0507 02:06:54.004486 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0507 02:06:54.004544 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:06:54.004717 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:06:54.004719 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:06:54.004722 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:06:54.004946 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:54.004949 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:54.005643 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:06:54.005644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:06:54.006190 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:54.006191 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:54.006251 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:54.006257 (+ 6us) client_negotiation.cc:769] Sending connection context
0507 02:06:54.006284 (+ 27us) client_negotiation.cc:240] Negotiation successful
0507 02:06:54.006318 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":64,"threads_started":1}
I20240507 02:06:54.006806 2135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:54.004426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.67:52629 (local address 127.31.131.66:43071)
0507 02:06:54.004465 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:54.004467 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:54.004612 (+ 145us) server_negotiation.cc:407] Connection header received
0507 02:06:54.004719 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:54.004721 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:54.004733 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:54.004801 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:06:54.005066 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:54.005512 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:54.006279 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:54.006707 (+ 428us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:54.006715 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:54.006734 (+ 19us) server_negotiation.cc:299] Negotiation successful
0507 02:06:54.006757 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20240507 02:06:54.006824 2129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:54.004404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.67:50533 (local address 127.31.131.65:33363)
0507 02:06:54.004454 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:54.004456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:54.004549 (+ 93us) server_negotiation.cc:407] Connection header received
0507 02:06:54.004599 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:54.004600 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:54.004610 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:54.004666 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:06:54.005010 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:54.005562 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:54.006267 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:54.006708 (+ 441us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:54.006716 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:54.006738 (+ 22us) server_negotiation.cc:299] Negotiation successful
0507 02:06:54.006765 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20240507 02:06:54.007068 1827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "415df9d71a1b4788a99362430ddfc8e5" candidate_uuid: "bc276328e6f24b5aa91552aa9fc95522" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af65885f5957407c938f4af9bc4f7bdc" is_pre_election: true
I20240507 02:06:54.007158 1827 raft_consensus.cc:2454] T 415df9d71a1b4788a99362430ddfc8e5 P af65885f5957407c938f4af9bc4f7bdc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc276328e6f24b5aa91552aa9fc95522 in term 0.
I20240507 02:06:54.007184 1687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "415df9d71a1b4788a99362430ddfc8e5" candidate_uuid: "bc276328e6f24b5aa91552aa9fc95522" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb01f5a060104233b45cb74c075401da" is_pre_election: true
I20240507 02:06:54.007256 1687 raft_consensus.cc:2454] T 415df9d71a1b4788a99362430ddfc8e5 P eb01f5a060104233b45cb74c075401da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc276328e6f24b5aa91552aa9fc95522 in term 0.
I20240507 02:06:54.007347 1899 leader_election.cc:302] T 415df9d71a1b4788a99362430ddfc8e5 P bc276328e6f24b5aa91552aa9fc95522 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af65885f5957407c938f4af9bc4f7bdc, bc276328e6f24b5aa91552aa9fc95522; no voters:
I20240507 02:06:54.007495 2127 raft_consensus.cc:2781] T 415df9d71a1b4788a99362430ddfc8e5 P bc276328e6f24b5aa91552aa9fc95522 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 02:06:54.007536 2127 raft_consensus.cc:482] T 415df9d71a1b4788a99362430ddfc8e5 P bc276328e6f24b5aa91552aa9fc95522 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 02:06:54.007555 2127 raft_consensus.cc:3037] T 415df9d71a1b4788a99362430ddfc8e5 P bc276328e6f24b5aa91552aa9fc95522 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:06:54.008155 2127 raft_consensus.cc:504] T 415df9d71a1b4788a99362430ddfc8e5 P bc276328e6f24b5aa91552aa9fc95522 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb01f5a060104233b45cb74c075401da" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33363 } } peers { permanent_uuid: "af65885f5957407c938f4af9bc4f7bdc" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 43071 } } peers { permanent_uuid: "bc276328e6f24b5aa91552aa9fc95522" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36899 } }
I20240507 02:06:54.008280 2127 leader_election.cc:288] T 415df9d71a1b4788a99362430ddfc8e5 P bc276328e6f24b5aa91552aa9fc95522 [CANDIDATE]: Term 1 election: Requested vote from peers eb01f5a060104233b45cb74c075401da (127.31.131.65:33363), af65885f5957407c938f4af9bc4f7bdc (127.31.131.66:43071)
I20240507 02:06:54.008399 1687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "415df9d71a1b4788a99362430ddfc8e5" candidate_uuid: "bc276328e6f24b5aa91552aa9fc95522" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb01f5a060104233b45cb74c075401da"
I20240507 02:06:54.008466 1687 raft_consensus.cc:3037] T 415df9d71a1b4788a99362430ddfc8e5 P eb01f5a060104233b45cb74c075401da [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:06:54.008435 1827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "415df9d71a1b4788a99362430ddfc8e5" candidate_uuid: "bc276328e6f24b5aa91552aa9fc95522" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af65885f5957407c938f4af9bc4f7bdc"
I20240507 02:06:54.008500 1827 raft_consensus.cc:3037] T 415df9d71a1b4788a99362430ddfc8e5 P af65885f5957407c938f4af9bc4f7bdc [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:06:54.009065 1687 raft_consensus.cc:2454] T 415df9d71a1b4788a99362430ddfc8e5 P eb01f5a060104233b45cb74c075401da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc276328e6f24b5aa91552aa9fc95522 in term 1.
I20240507 02:06:54.009070 1827 raft_consensus.cc:2454] T 415df9d71a1b4788a99362430ddfc8e5 P af65885f5957407c938f4af9bc4f7bdc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc276328e6f24b5aa91552aa9fc95522 in term 1.
I20240507 02:06:54.009220 1902 leader_election.cc:302] T 415df9d71a1b4788a99362430ddfc8e5 P bc276328e6f24b5aa91552aa9fc95522 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc276328e6f24b5aa91552aa9fc95522, eb01f5a060104233b45cb74c075401da; no voters:
I20240507 02:06:54.009462 2127 raft_consensus.cc:2781] T 415df9d71a1b4788a99362430ddfc8e5 P bc276328e6f24b5aa91552aa9fc95522 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:06:54.009624 2127 raft_consensus.cc:686] T 415df9d71a1b4788a99362430ddfc8e5 P bc276328e6f24b5aa91552aa9fc95522 [term 1 LEADER]: Becoming Leader. State: Replica: bc276328e6f24b5aa91552aa9fc95522, State: Running, Role: LEADER
I20240507 02:06:54.009704 2127 consensus_queue.cc:237] T 415df9d71a1b4788a99362430ddfc8e5 P bc276328e6f24b5aa91552aa9fc95522 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb01f5a060104233b45cb74c075401da" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33363 } } peers { permanent_uuid: "af65885f5957407c938f4af9bc4f7bdc" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 43071 } } peers { permanent_uuid: "bc276328e6f24b5aa91552aa9fc95522" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36899 } }
I20240507 02:06:54.010311 1528 catalog_manager.cc:5484] T 415df9d71a1b4788a99362430ddfc8e5 P bc276328e6f24b5aa91552aa9fc95522 reported cstate change: term changed from 0 to 1, leader changed from <none> to bc276328e6f24b5aa91552aa9fc95522 (127.31.131.67). New cstate: current_term: 1 leader_uuid: "bc276328e6f24b5aa91552aa9fc95522" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb01f5a060104233b45cb74c075401da" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af65885f5957407c938f4af9bc4f7bdc" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 43071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc276328e6f24b5aa91552aa9fc95522" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36899 } health_report { overall_health: HEALTHY } } }
I20240507 02:06:54.059868 2131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:54.058054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35156 (local address 127.31.131.67:36899)
0507 02:06:54.058109 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:54.058110 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:54.058119 (+ 9us) server_negotiation.cc:407] Connection header received
0507 02:06:54.058163 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:54.058163 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:54.058180 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:54.058250 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0507 02:06:54.058443 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:54.058952 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:54.059431 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:54.059548 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:54.059585 (+ 37us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0507 02:06:54.059673 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0507 02:06:54.059735 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:54.059772 (+ 37us) server_negotiation.cc:299] Negotiation successful
0507 02:06:54.059790 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20240507 02:06:54.063967 2129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:54.062121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.67:33091 (local address 127.31.131.65:33363)
0507 02:06:54.062182 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:54.062184 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:54.062216 (+ 32us) server_negotiation.cc:407] Connection header received
0507 02:06:54.062264 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:54.062264 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:54.062272 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:54.062313 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0507 02:06:54.062580 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:54.063030 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:54.063582 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:54.063723 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:54.063739 (+ 16us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0507 02:06:54.063761 (+ 22us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0507 02:06:54.063817 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:54.063876 (+ 59us) server_negotiation.cc:299] Negotiation successful
0507 02:06:54.063895 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20240507 02:06:54.064110 2155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:54.061966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.65:33363 (local address 127.31.131.67:33091)
0507 02:06:54.062166 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0507 02:06:54.062175 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0507 02:06:54.062228 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:06:54.062326 (+ 98us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:06:54.062326 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:06:54.062328 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0507 02:06:54.062485 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:54.062486 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:54.063104 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:06:54.063105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:06:54.063516 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:54.063518 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:54.063574 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:54.063590 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0507 02:06:54.063829 (+ 239us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0507 02:06:54.063832 (+ 3us) client_negotiation.cc:769] Sending connection context
0507 02:06:54.063867 (+ 35us) client_negotiation.cc:240] Negotiation successful
0507 02:06:54.063911 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":79,"threads_started":1}
I20240507 02:06:54.065824 1827 raft_consensus.cc:1261] T 415df9d71a1b4788a99362430ddfc8e5 P af65885f5957407c938f4af9bc4f7bdc [term 1 FOLLOWER]: Refusing update from remote peer bc276328e6f24b5aa91552aa9fc95522: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 02:06:54.065897 1687 raft_consensus.cc:1261] T 415df9d71a1b4788a99362430ddfc8e5 P eb01f5a060104233b45cb74c075401da [term 1 FOLLOWER]: Refusing update from remote peer bc276328e6f24b5aa91552aa9fc95522: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 02:06:54.066058 2127 consensus_queue.cc:1035] T 415df9d71a1b4788a99362430ddfc8e5 P bc276328e6f24b5aa91552aa9fc95522 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af65885f5957407c938f4af9bc4f7bdc" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 43071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 02:06:54.066177 2151 consensus_queue.cc:1035] T 415df9d71a1b4788a99362430ddfc8e5 P bc276328e6f24b5aa91552aa9fc95522 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb01f5a060104233b45cb74c075401da" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 02:06:54.072322 2143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:54.070523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.67:36899 (local address 127.31.131.65:39905)
0507 02:06:54.070576 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0507 02:06:54.070587 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0507 02:06:54.070625 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:06:54.070739 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:06:54.070739 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:06:54.070740 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0507 02:06:54.070925 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:54.070927 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:54.071474 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:06:54.071475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:06:54.071866 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:54.071867 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:54.071922 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:54.071927 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0507 02:06:54.072170 (+ 243us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0507 02:06:54.072173 (+ 3us) client_negotiation.cc:769] Sending connection context
0507 02:06:54.072212 (+ 39us) client_negotiation.cc:240] Negotiation successful
0507 02:06:54.072258 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20240507 02:06:54.072319 2131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:54.070598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:39905 (local address 127.31.131.67:36899)
0507 02:06:54.070626 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:54.070627 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:54.070635 (+ 8us) server_negotiation.cc:407] Connection header received
0507 02:06:54.070680 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:54.070680 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:54.070688 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:54.070735 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0507 02:06:54.070984 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:54.071413 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:54.071931 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:54.072075 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:54.072094 (+ 19us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0507 02:06:54.072114 (+ 20us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0507 02:06:54.072160 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:54.072221 (+ 61us) server_negotiation.cc:299] Negotiation successful
0507 02:06:54.072238 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
W20240507 02:06:54.072696 2093 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20240507 02:06:55.590459 1528 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34374:
table { table_name: "test-table" } modify_external_catalogs: true
I20240507 02:06:55.590569 1528 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34374:
table { table_name: "test-table" } modify_external_catalogs: true
I20240507 02:06:55.591323 1528 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 27251517e7da4be8bebf6abfb08c2a57: Sending DeleteTablet for 3 replicas of tablet 415df9d71a1b4788a99362430ddfc8e5
I20240507 02:06:55.591595 1666 tablet_service.cc:1514] Processing DeleteTablet for tablet 415df9d71a1b4788a99362430ddfc8e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:06:55 UTC) from {username='kudu', principal='kudu/127.31.131.126@KRBTEST.COM'} at 127.0.0.1:51460
I20240507 02:06:55.591622 1948 tablet_service.cc:1514] Processing DeleteTablet for tablet 415df9d71a1b4788a99362430ddfc8e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:06:55 UTC) from {username='kudu', principal='kudu/127.31.131.126@KRBTEST.COM'} at 127.0.0.1:47746
I20240507 02:06:55.591684 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 1601
I20240507 02:06:55.591779 1808 tablet_service.cc:1514] Processing DeleteTablet for tablet 415df9d71a1b4788a99362430ddfc8e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:06:55 UTC) from {username='kudu', principal='kudu/127.31.131.126@KRBTEST.COM'} at 127.0.0.1:43084
I20240507 02:06:55.593051 2182 tablet_replica.cc:330] T 415df9d71a1b4788a99362430ddfc8e5 P af65885f5957407c938f4af9bc4f7bdc: stopping tablet replica
I20240507 02:06:55.593106 2182 raft_consensus.cc:2229] T 415df9d71a1b4788a99362430ddfc8e5 P af65885f5957407c938f4af9bc4f7bdc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 02:06:55.593132 2182 raft_consensus.cc:2258] T 415df9d71a1b4788a99362430ddfc8e5 P af65885f5957407c938f4af9bc4f7bdc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 02:06:55.593323 2182 ts_tablet_manager.cc:1875] T 415df9d71a1b4788a99362430ddfc8e5 P af65885f5957407c938f4af9bc4f7bdc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 02:06:55.593573 2181 tablet_replica.cc:330] T 415df9d71a1b4788a99362430ddfc8e5 P bc276328e6f24b5aa91552aa9fc95522: stopping tablet replica
I20240507 02:06:55.593626 2181 raft_consensus.cc:2229] T 415df9d71a1b4788a99362430ddfc8e5 P bc276328e6f24b5aa91552aa9fc95522 [term 1 LEADER]: Raft consensus shutting down.
I20240507 02:06:55.593714 2181 raft_consensus.cc:2258] T 415df9d71a1b4788a99362430ddfc8e5 P bc276328e6f24b5aa91552aa9fc95522 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 02:06:55.593935 2181 ts_tablet_manager.cc:1875] T 415df9d71a1b4788a99362430ddfc8e5 P bc276328e6f24b5aa91552aa9fc95522: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 02:06:55.594877 2182 ts_tablet_manager.cc:1888] T 415df9d71a1b4788a99362430ddfc8e5 P af65885f5957407c938f4af9bc4f7bdc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20240507 02:06:55.594928 2182 log.cc:1196] T 415df9d71a1b4788a99362430ddfc8e5 P af65885f5957407c938f4af9bc4f7bdc: Deleting WAL directory at /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/ts-1/wal/wals/415df9d71a1b4788a99362430ddfc8e5
I20240507 02:06:55.595145 2182 ts_tablet_manager.cc:1909] T 415df9d71a1b4788a99362430ddfc8e5 P af65885f5957407c938f4af9bc4f7bdc: Deleting consensus metadata
I20240507 02:06:55.595422 1513 catalog_manager.cc:4830] TS af65885f5957407c938f4af9bc4f7bdc (127.31.131.66:43071): tablet 415df9d71a1b4788a99362430ddfc8e5 (table test-table [id=10b246ad68aa47d6a30d4be4e4706ab4]) successfully deleted
I20240507 02:06:55.595432 2181 ts_tablet_manager.cc:1888] T 415df9d71a1b4788a99362430ddfc8e5 P bc276328e6f24b5aa91552aa9fc95522: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20240507 02:06:55.595479 2181 log.cc:1196] T 415df9d71a1b4788a99362430ddfc8e5 P bc276328e6f24b5aa91552aa9fc95522: Deleting WAL directory at /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1715047604443657-32269-0/minicluster-data/ts-2/wal/wals/415df9d71a1b4788a99362430ddfc8e5
I20240507 02:06:55.595666 2181 ts_tablet_manager.cc:1909] T 415df9d71a1b4788a99362430ddfc8e5 P bc276328e6f24b5aa91552aa9fc95522: Deleting consensus metadata
I20240507 02:06:55.596004 1516 catalog_manager.cc:4830] TS bc276328e6f24b5aa91552aa9fc95522 (127.31.131.67:36899): tablet 415df9d71a1b4788a99362430ddfc8e5 (table test-table [id=10b246ad68aa47d6a30d4be4e4706ab4]) successfully deleted
W20240507 02:06:55.598064 1516 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.131.65:33363 (error 108)
I20240507 02:06:55.598271 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 1742
I20240507 02:06:55.598886 2184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:55.598407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.65:33363 (local address 127.0.0.1:58910)
0507 02:06:55.598817 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0507 02:06:55.598851 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.65:33363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":49,"threads_started":1}
W20240507 02:06:55.598980 1516 catalog_manager.cc:4573] TS eb01f5a060104233b45cb74c075401da (127.31.131.65:33363): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 415df9d71a1b4788a99362430ddfc8e5: Network error: Client connection negotiation failed: client connection to 127.31.131.65:33363: connect: Connection refused (error 111)
I20240507 02:06:55.603633 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 1883
I20240507 02:06:55.608464 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 1497
2024-05-07T02:06:55Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5765 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 07 02:06:55 dist-test-slave-v0bg krb5kdc[2187](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 07 02:06:55 dist-test-slave-v0bg krb5kdc[2187](info): set up 2 sockets
May 07 02:06:55 dist-test-slave-v0bg krb5kdc[2187](info): commencing operation
krb5kdc: starting...
W20240507 02:06:56.499292 32269 mini_kdc.cc:121] Time spent starting KDC: real 0.872s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/krb5kdc/test-user.keytab.
May 07 02:06:56 dist-test-slave-v0bg krb5kdc[2187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-07T02:06:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-07T02:06:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.131.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab.
Entry for principal kudu/127.31.131.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab.
I20240507 02:06:56.602854 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:44827
--webserver_interface=127.31.131.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab
--principal=kudu/127.31.131.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:37993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.131.126:44827
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:06:56.678457 2200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:06:56.678663 2200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:06:56.678689 2200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:06:56.679911 2200 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240507 02:06:56.679946 2200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:06:56.679957 2200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:06:56.679967 2200 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240507 02:06:56.679977 2200 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240507 02:06:56.681434 2200 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:37993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.131.126:44827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab
--principal=kudu/127.31.131.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:44827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.131.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:06:56.681591 2200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:06:56.681733 2200 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:06:56.685290 2205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:06:56.685333 2206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:06:56.685405 2200 server_base.cc:1034] running on GCE node
W20240507 02:06:56.685571 2208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:06:56.685815 2200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:06:56.686049 2200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:06:56.687176 2200 hybrid_clock.cc:648] HybridClock initialized: now 1715047616687159 us; error 30 us; skew 500 ppm
May 07 02:06:56 dist-test-slave-v0bg krb5kdc[2187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047616, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:06:56.689370 2200 init.cc:377] Logged in from keytab as kudu/127.31.131.126@KRBTEST.COM (short username kudu)
I20240507 02:06:56.689765 2200 webserver.cc:460] Webserver started at http://127.31.131.126:39959/ using document root <none> and password file <none>
I20240507 02:06:56.689924 2200 fs_manager.cc:359] Metadata directory not provided
I20240507 02:06:56.689966 2200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:06:56.690071 2200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:06:56.691053 2200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/minicluster-data/master-0/data/instance:
uuid: "20156a3dafd64620bd312f1c6f79cf12"
format_stamp: "Formatted at 2024-05-07 02:06:56 on dist-test-slave-v0bg"
server_key: "751752fbaebc464fa869eac38e7d9a48"
server_key_iv: "83bb6220811147df9bbb0afa181d108c"
server_key_version: "encryptionkey@0"
I20240507 02:06:56.691357 2200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/minicluster-data/master-0/wal/instance:
uuid: "20156a3dafd64620bd312f1c6f79cf12"
format_stamp: "Formatted at 2024-05-07 02:06:56 on dist-test-slave-v0bg"
server_key: "751752fbaebc464fa869eac38e7d9a48"
server_key_iv: "83bb6220811147df9bbb0afa181d108c"
server_key_version: "encryptionkey@0"
I20240507 02:06:56.692440 2200 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240507 02:06:56.693032 2215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:56.693177 2200 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240507 02:06:56.693249 2200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/minicluster-data/master-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/minicluster-data/master-0/wal
uuid: "20156a3dafd64620bd312f1c6f79cf12"
format_stamp: "Formatted at 2024-05-07 02:06:56 on dist-test-slave-v0bg"
server_key: "751752fbaebc464fa869eac38e7d9a48"
server_key_iv: "83bb6220811147df9bbb0afa181d108c"
server_key_version: "encryptionkey@0"
I20240507 02:06:56.693310 2200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:06:56.710109 2200 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:06:56.712687 2200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:06:56.712862 2200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:06:56.716606 2200 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.126:44827
I20240507 02:06:56.716645 2267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.126:44827 every 8 connection(s)
I20240507 02:06:56.716948 2200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
I20240507 02:06:56.717675 2268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:06:56.719691 2268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20156a3dafd64620bd312f1c6f79cf12: Bootstrap starting.
I20240507 02:06:56.720173 2268 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20156a3dafd64620bd312f1c6f79cf12: Neither blocks nor log segments found. Creating new log.
I20240507 02:06:56.720405 2268 log.cc:824] T 00000000000000000000000000000000 P 20156a3dafd64620bd312f1c6f79cf12: Log is configured to *not* fsync() on all Append() calls
I20240507 02:06:56.720990 2268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20156a3dafd64620bd312f1c6f79cf12: No bootstrap required, opened a new log
I20240507 02:06:56.722302 2268 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20156a3dafd64620bd312f1c6f79cf12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20156a3dafd64620bd312f1c6f79cf12" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 44827 } }
I20240507 02:06:56.722414 2268 raft_consensus.cc:374] T 00000000000000000000000000000000 P 20156a3dafd64620bd312f1c6f79cf12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:06:56.722436 2268 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20156a3dafd64620bd312f1c6f79cf12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20156a3dafd64620bd312f1c6f79cf12, State: Initialized, Role: FOLLOWER
I20240507 02:06:56.722539 2268 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20156a3dafd64620bd312f1c6f79cf12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20156a3dafd64620bd312f1c6f79cf12" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 44827 } }
I20240507 02:06:56.722631 2268 raft_consensus.cc:388] T 00000000000000000000000000000000 P 20156a3dafd64620bd312f1c6f79cf12 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 02:06:56.722658 2268 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20156a3dafd64620bd312f1c6f79cf12 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 02:06:56.722700 2268 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20156a3dafd64620bd312f1c6f79cf12 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:06:56.723160 2268 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20156a3dafd64620bd312f1c6f79cf12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20156a3dafd64620bd312f1c6f79cf12" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 44827 } }
I20240507 02:06:56.723258 2268 leader_election.cc:302] T 00000000000000000000000000000000 P 20156a3dafd64620bd312f1c6f79cf12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 20156a3dafd64620bd312f1c6f79cf12; no voters:
I20240507 02:06:56.723381 2268 leader_election.cc:288] T 00000000000000000000000000000000 P 20156a3dafd64620bd312f1c6f79cf12 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240507 02:06:56.723433 2271 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20156a3dafd64620bd312f1c6f79cf12 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:06:56.723589 2268 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20156a3dafd64620bd312f1c6f79cf12 [sys.catalog]: configured and running, proceeding with master startup.
I20240507 02:06:56.723852 2271 raft_consensus.cc:686] T 00000000000000000000000000000000 P 20156a3dafd64620bd312f1c6f79cf12 [term 1 LEADER]: Becoming Leader. State: Replica: 20156a3dafd64620bd312f1c6f79cf12, State: Running, Role: LEADER
I20240507 02:06:56.723937 2271 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20156a3dafd64620bd312f1c6f79cf12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20156a3dafd64620bd312f1c6f79cf12" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 44827 } }
I20240507 02:06:56.724274 2272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20156a3dafd64620bd312f1c6f79cf12 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "20156a3dafd64620bd312f1c6f79cf12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20156a3dafd64620bd312f1c6f79cf12" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 44827 } } }
I20240507 02:06:56.724303 2273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20156a3dafd64620bd312f1c6f79cf12 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20156a3dafd64620bd312f1c6f79cf12. Latest consensus state: current_term: 1 leader_uuid: "20156a3dafd64620bd312f1c6f79cf12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20156a3dafd64620bd312f1c6f79cf12" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 44827 } } }
I20240507 02:06:56.724362 2272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20156a3dafd64620bd312f1c6f79cf12 [sys.catalog]: This master's current role is: LEADER
I20240507 02:06:56.724373 2273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20156a3dafd64620bd312f1c6f79cf12 [sys.catalog]: This master's current role is: LEADER
I20240507 02:06:56.724581 2280 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 02:06:56.725175 2280 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240507 02:06:56.726194 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 2200
I20240507 02:06:56.726260 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/minicluster-data/master-0/wal/instance
I20240507 02:06:56.726367 32269 external_mini_cluster.cc:1428] Setting key 5f3d78d184966c658243c0e9a457b062
I20240507 02:06:56.727097 2280 catalog_manager.cc:1347] Generated new cluster ID: 6d58a5d80c444499b460fff9b40efe33
I20240507 02:06:56.727145 2280 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
May 07 02:06:56 dist-test-slave-v0bg krb5kdc[2187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:06:56.736959 2291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:56.726779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36308 (local address 127.31.131.126:44827)
0507 02:06:56.726927 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:56.726938 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:56.726965 (+ 27us) server_negotiation.cc:407] Connection header received
0507 02:06:56.727090 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:56.727096 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:56.727164 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:56.727306 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:06:56.728414 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:56.729562 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:56.730568 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:56.730805 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:56.733031 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:06:56.733037 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:06:56.733037 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:06:56.733056 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:06:56.735611 (+ 2555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:56.735940 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:56.735943 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:56.735944 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:56.736004 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:56.736149 (+ 145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:56.736150 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:56.736151 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:56.736383 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:06:56.736522 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:56.736628 (+ 106us) server_negotiation.cc:299] Negotiation successful
0507 02:06:56.736755 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":78,"threads_started":1}
I20240507 02:06:56.753021 2280 catalog_manager.cc:1370] Generated new certificate authority record
I20240507 02:06:56.753533 2280 catalog_manager.cc:1504] Loading token signing keys...
I20240507 02:06:56.758299 2280 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 20156a3dafd64620bd312f1c6f79cf12: Generated new TSK 0
I20240507 02:06:56.758445 2280 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.131.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab.
Entry for principal kudu/127.31.131.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab.
I20240507 02:06:56.787298 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.65:0
--local_ip_for_outbound_sockets=127.31.131.65
--webserver_interface=127.31.131.65
--webserver_port=0
--tserver_master_addrs=127.31.131.126:44827
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab
--principal=kudu/127.31.131.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:37993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:06:56.861600 2294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:06:56.861761 2294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:06:56.861789 2294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:06:56.863032 2294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:06:56.863072 2294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:06:56.863103 2294 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.65
I20240507 02:06:56.864616 2294 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:37993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab
--principal=kudu/127.31.131.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.131.65
--webserver_port=0
--tserver_master_addrs=127.31.131.126:44827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.65
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:06:56.864813 2294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:06:56.864979 2294 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240507 02:06:56.869196 2294 server_base.cc:1034] running on GCE node
W20240507 02:06:56.869179 2300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:06:56.869164 2299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:06:56.869509 2302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:06:56.869689 2294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:06:56.869941 2294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:06:56.871078 2294 hybrid_clock.cc:648] HybridClock initialized: now 1715047616871062 us; error 31 us; skew 500 ppm
May 07 02:06:56 dist-test-slave-v0bg krb5kdc[2187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047616, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:06:56.873628 2294 init.cc:377] Logged in from keytab as kudu/127.31.131.65@KRBTEST.COM (short username kudu)
I20240507 02:06:56.874068 2294 webserver.cc:460] Webserver started at http://127.31.131.65:34959/ using document root <none> and password file <none>
I20240507 02:06:56.874253 2294 fs_manager.cc:359] Metadata directory not provided
I20240507 02:06:56.874300 2294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:06:56.874406 2294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:06:56.875591 2294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-0/data/instance:
uuid: "2e53a168ffb5443bbf28454f7effc922"
format_stamp: "Formatted at 2024-05-07 02:06:56 on dist-test-slave-v0bg"
server_key: "47481cbe5064cbdb21b99a81a0041c0d"
server_key_iv: "8c7d86cf24325a2a6caa92b487762e19"
server_key_version: "encryptionkey@0"
I20240507 02:06:56.875906 2294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-0/wal/instance:
uuid: "2e53a168ffb5443bbf28454f7effc922"
format_stamp: "Formatted at 2024-05-07 02:06:56 on dist-test-slave-v0bg"
server_key: "47481cbe5064cbdb21b99a81a0041c0d"
server_key_iv: "8c7d86cf24325a2a6caa92b487762e19"
server_key_version: "encryptionkey@0"
I20240507 02:06:56.877135 2294 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240507 02:06:56.877864 2309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:56.878039 2294 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240507 02:06:56.878098 2294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-0/wal
uuid: "2e53a168ffb5443bbf28454f7effc922"
format_stamp: "Formatted at 2024-05-07 02:06:56 on dist-test-slave-v0bg"
server_key: "47481cbe5064cbdb21b99a81a0041c0d"
server_key_iv: "8c7d86cf24325a2a6caa92b487762e19"
server_key_version: "encryptionkey@0"
I20240507 02:06:56.878156 2294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:06:56.901705 2294 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:06:56.903640 2294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:06:56.903750 2294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:06:56.903936 2294 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:06:56.904204 2294 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:06:56.904234 2294 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:56.904254 2294 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:06:56.904278 2294 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:56.909004 2294 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.65:41609
I20240507 02:06:56.909070 2423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.65:41609 every 8 connection(s)
I20240507 02:06:56.909314 2294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
I20240507 02:06:56.911233 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 2294
I20240507 02:06:56.911310 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-0/wal/instance
I20240507 02:06:56.911449 32269 external_mini_cluster.cc:1428] Setting key 6d6236947a4ee1f10b93b0ab8a2e3627
May 07 02:06:56 dist-test-slave-v0bg krb5kdc[2187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047616, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.65@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:06:56.917228 2291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:56.909955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:58909 (local address 127.31.131.126:44827)
0507 02:06:56.910018 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:56.910020 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:56.910172 (+ 152us) server_negotiation.cc:407] Connection header received
0507 02:06:56.910617 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:56.910618 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:56.910632 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:56.910689 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:06:56.911417 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:56.912039 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:56.912563 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:56.912742 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:56.914891 (+ 2149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:06:56.914895 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:06:56.914895 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:06:56.914901 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:06:56.916099 (+ 1198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:56.916502 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:56.916502 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:56.916503 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:56.916519 (+ 16us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:56.916696 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:56.916697 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:56.916697 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:56.916781 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:06:56.916821 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:56.917098 (+ 277us) server_negotiation.cc:299] Negotiation successful
0507 02:06:56.917159 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20240507 02:06:56.917404 2426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:56.909905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:44827 (local address 127.31.131.65:58909)
0507 02:06:56.910068 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0507 02:06:56.910096 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0507 02:06:56.910521 (+ 425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:06:56.910743 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:06:56.910747 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:06:56.911008 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:06:56.911323 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:56.911328 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:56.912112 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:06:56.912113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:06:56.912487 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:56.912488 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:56.912838 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:56.913309 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:06:56.913329 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:06:56.914787 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:06:56.916179 (+ 1392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:06:56.916185 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:06:56.916188 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:06:56.916434 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:06:56.916569 (+ 135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:06:56.916570 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:06:56.916571 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:06:56.916645 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:06:56.916828 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:06:56.916832 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:06:56.916973 (+ 141us) client_negotiation.cc:769] Sending connection context
0507 02:06:56.917093 (+ 120us) client_negotiation.cc:240] Negotiation successful
0507 02:06:56.917220 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":91,"threads_started":1}
I20240507 02:06:56.917879 2424 heartbeater.cc:347] Connected to a master server at 127.31.131.126:44827
I20240507 02:06:56.917963 2424 heartbeater.cc:464] Registering TS with master...
I20240507 02:06:56.918131 2424 heartbeater.cc:511] Master 127.31.131.126:44827 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.31.131.66@KRBTEST.COM; defaulting to no policy
I20240507 02:06:56.918638 2232 ts_manager.cc:194] Registered new tserver with Master: 2e53a168ffb5443bbf28454f7effc922 (127.31.131.65:41609)
I20240507 02:06:56.919306 2232 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.131.65@KRBTEST.COM'} at 127.31.131.65:58909
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab.
Entry for principal kudu/127.31.131.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab.
I20240507 02:06:56.933455 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.66:0
--local_ip_for_outbound_sockets=127.31.131.66
--webserver_interface=127.31.131.66
--webserver_port=0
--tserver_master_addrs=127.31.131.126:44827
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab
--principal=kudu/127.31.131.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:37993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:06:57.009519 2429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:06:57.009699 2429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:06:57.009732 2429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:06:57.011011 2429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:06:57.011058 2429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:06:57.011093 2429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.66
I20240507 02:06:57.012635 2429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:37993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab
--principal=kudu/127.31.131.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.131.66
--webserver_port=0
--tserver_master_addrs=127.31.131.126:44827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.66
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:06:57.012948 2429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:06:57.013136 2429 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:06:57.016832 2434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:06:57.016882 2435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:06:57.016990 2437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:06:57.017041 2429 server_base.cc:1034] running on GCE node
I20240507 02:06:57.017203 2429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:06:57.017397 2429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:06:57.018525 2429 hybrid_clock.cc:648] HybridClock initialized: now 1715047617018511 us; error 29 us; skew 500 ppm
May 07 02:06:57 dist-test-slave-v0bg krb5kdc[2187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047617, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:06:57.020649 2429 init.cc:377] Logged in from keytab as kudu/127.31.131.66@KRBTEST.COM (short username kudu)
I20240507 02:06:57.021039 2429 webserver.cc:460] Webserver started at http://127.31.131.66:44063/ using document root <none> and password file <none>
I20240507 02:06:57.021196 2429 fs_manager.cc:359] Metadata directory not provided
I20240507 02:06:57.021235 2429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:06:57.021332 2429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:06:57.022221 2429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-1/data/instance:
uuid: "c1ac2d645600477fae9aa1a53a3b0fdf"
format_stamp: "Formatted at 2024-05-07 02:06:57 on dist-test-slave-v0bg"
server_key: "76a3538b959bee4b18edbbc84497a2ac"
server_key_iv: "c740be9bfd9aadec3f8f55575879a0c4"
server_key_version: "encryptionkey@0"
I20240507 02:06:57.022492 2429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-1/wal/instance:
uuid: "c1ac2d645600477fae9aa1a53a3b0fdf"
format_stamp: "Formatted at 2024-05-07 02:06:57 on dist-test-slave-v0bg"
server_key: "76a3538b959bee4b18edbbc84497a2ac"
server_key_iv: "c740be9bfd9aadec3f8f55575879a0c4"
server_key_version: "encryptionkey@0"
I20240507 02:06:57.023779 2429 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240507 02:06:57.024374 2444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:57.024513 2429 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240507 02:06:57.024583 2429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-1/wal
uuid: "c1ac2d645600477fae9aa1a53a3b0fdf"
format_stamp: "Formatted at 2024-05-07 02:06:57 on dist-test-slave-v0bg"
server_key: "76a3538b959bee4b18edbbc84497a2ac"
server_key_iv: "c740be9bfd9aadec3f8f55575879a0c4"
server_key_version: "encryptionkey@0"
I20240507 02:06:57.024643 2429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:06:57.035434 2429 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:06:57.036805 2429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:06:57.036906 2429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:06:57.037061 2429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:06:57.037324 2429 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:06:57.037354 2429 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:57.037380 2429 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:06:57.037397 2429 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:57.042039 2429 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.66:33119
I20240507 02:06:57.042091 2558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.66:33119 every 8 connection(s)
I20240507 02:06:57.042321 2429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
May 07 02:06:57 dist-test-slave-v0bg krb5kdc[2187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047617, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.66@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:06:57.046963 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 2429
I20240507 02:06:57.047045 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-1/wal/instance
I20240507 02:06:57.047178 32269 external_mini_cluster.cc:1428] Setting key 5c8979a1bfb1c46132c791e26ebd8886
I20240507 02:06:57.049748 2291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:57.042916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.66:40287 (local address 127.31.131.126:44827)
0507 02:06:57.042977 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:57.042979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:57.043125 (+ 146us) server_negotiation.cc:407] Connection header received
0507 02:06:57.043549 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:57.043550 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:57.043560 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:57.043615 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:06:57.044267 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:57.044739 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:57.045208 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:57.045311 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:57.047231 (+ 1920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:06:57.047235 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:06:57.047235 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:06:57.047241 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:06:57.048657 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:57.048962 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:57.048963 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:57.048964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:57.048987 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:57.049185 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:57.049186 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:57.049186 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:57.049270 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:06:57.049315 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:57.049573 (+ 258us) server_negotiation.cc:299] Negotiation successful
0507 02:06:57.049650 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20240507 02:06:57.049821 2561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:57.042895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:44827 (local address 127.31.131.66:40287)
0507 02:06:57.043026 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0507 02:06:57.043052 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0507 02:06:57.043473 (+ 421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:06:57.043660 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:06:57.043664 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:06:57.043924 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:06:57.044194 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:57.044197 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:57.044796 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:06:57.044797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:06:57.045155 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:57.045156 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:57.045435 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:57.045854 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:06:57.045866 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:06:57.047135 (+ 1269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:06:57.048727 (+ 1592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:06:57.048732 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:06:57.048734 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:06:57.048904 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:06:57.049052 (+ 148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:06:57.049053 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:06:57.049053 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:06:57.049125 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:06:57.049320 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:06:57.049323 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:06:57.049449 (+ 126us) client_negotiation.cc:769] Sending connection context
0507 02:06:57.049562 (+ 113us) client_negotiation.cc:240] Negotiation successful
0507 02:06:57.049672 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":74,"threads_started":1}
I20240507 02:06:57.050264 2559 heartbeater.cc:347] Connected to a master server at 127.31.131.126:44827
I20240507 02:06:57.050349 2559 heartbeater.cc:464] Registering TS with master...
I20240507 02:06:57.050518 2559 heartbeater.cc:511] Master 127.31.131.126:44827 requested a full tablet report, sending...
I20240507 02:06:57.050907 2232 ts_manager.cc:194] Registered new tserver with Master: c1ac2d645600477fae9aa1a53a3b0fdf (127.31.131.66:33119)
I20240507 02:06:57.051261 2232 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.131.66@KRBTEST.COM'} at 127.31.131.66:40287
WARNING: no policy specified for kudu/127.31.131.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab.
Entry for principal kudu/127.31.131.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab.
I20240507 02:06:57.067255 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.67:0
--local_ip_for_outbound_sockets=127.31.131.67
--webserver_interface=127.31.131.67
--webserver_port=0
--tserver_master_addrs=127.31.131.126:44827
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab
--principal=kudu/127.31.131.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:37993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:06:57.140784 2564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:06:57.140926 2564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:06:57.140946 2564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:06:57.142156 2564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:06:57.142194 2564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:06:57.142223 2564 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.67
I20240507 02:06:57.143759 2564 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:37993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab
--principal=kudu/127.31.131.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.131.67
--webserver_port=0
--tserver_master_addrs=127.31.131.126:44827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.67
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:06:57.143939 2564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:06:57.144096 2564 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:06:57.147611 2570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:06:57.147622 2572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:06:57.147745 2569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:06:57.147792 2564 server_base.cc:1034] running on GCE node
I20240507 02:06:57.147946 2564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:06:57.148105 2564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:06:57.149231 2564 hybrid_clock.cc:648] HybridClock initialized: now 1715047617149222 us; error 29 us; skew 500 ppm
May 07 02:06:57 dist-test-slave-v0bg krb5kdc[2187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047617, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:06:57.151347 2564 init.cc:377] Logged in from keytab as kudu/127.31.131.67@KRBTEST.COM (short username kudu)
I20240507 02:06:57.151707 2564 webserver.cc:460] Webserver started at http://127.31.131.67:45495/ using document root <none> and password file <none>
I20240507 02:06:57.151875 2564 fs_manager.cc:359] Metadata directory not provided
I20240507 02:06:57.151912 2564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:06:57.152007 2564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:06:57.152875 2564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-2/data/instance:
uuid: "3f1ef547e5394c1da3274eafad360a88"
format_stamp: "Formatted at 2024-05-07 02:06:57 on dist-test-slave-v0bg"
server_key: "a773a4146e36b39b16f1083318891daf"
server_key_iv: "66ba85cac8ea225404bdd938d9853ae7"
server_key_version: "encryptionkey@0"
I20240507 02:06:57.153137 2564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-2/wal/instance:
uuid: "3f1ef547e5394c1da3274eafad360a88"
format_stamp: "Formatted at 2024-05-07 02:06:57 on dist-test-slave-v0bg"
server_key: "a773a4146e36b39b16f1083318891daf"
server_key_iv: "66ba85cac8ea225404bdd938d9853ae7"
server_key_version: "encryptionkey@0"
I20240507 02:06:57.154140 2564 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240507 02:06:57.154700 2579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:57.154840 2564 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240507 02:06:57.154906 2564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-2/wal
uuid: "3f1ef547e5394c1da3274eafad360a88"
format_stamp: "Formatted at 2024-05-07 02:06:57 on dist-test-slave-v0bg"
server_key: "a773a4146e36b39b16f1083318891daf"
server_key_iv: "66ba85cac8ea225404bdd938d9853ae7"
server_key_version: "encryptionkey@0"
I20240507 02:06:57.154961 2564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:06:57.165427 2564 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:06:57.166936 2564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:06:57.167032 2564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:06:57.167188 2564 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:06:57.167439 2564 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:06:57.167469 2564 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:57.167495 2564 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:06:57.167516 2564 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:57.172235 2564 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.67:45841
I20240507 02:06:57.172293 2693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.67:45841 every 8 connection(s)
I20240507 02:06:57.172523 2564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
May 07 02:06:57 dist-test-slave-v0bg krb5kdc[2187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047617, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.67@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:06:57.179317 2291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:57.173107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.67:48361 (local address 127.31.131.126:44827)
0507 02:06:57.173173 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:57.173175 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:57.173314 (+ 139us) server_negotiation.cc:407] Connection header received
0507 02:06:57.173749 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:57.173750 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:57.173759 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:57.173813 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:06:57.174454 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:57.174963 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:57.175444 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:57.175542 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:57.177405 (+ 1863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:06:57.177409 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:06:57.177410 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:06:57.177416 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:06:57.178438 (+ 1022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:57.178756 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:57.178757 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:57.178757 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:57.178776 (+ 19us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:57.178893 (+ 117us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:57.178893 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:57.178893 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:57.178969 (+ 76us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:06:57.179009 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:57.179185 (+ 176us) server_negotiation.cc:299] Negotiation successful
0507 02:06:57.179254 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20240507 02:06:57.179405 2696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:57.173081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:44827 (local address 127.31.131.67:48361)
0507 02:06:57.173210 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0507 02:06:57.173236 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0507 02:06:57.173664 (+ 428us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:06:57.173857 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:06:57.173865 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:06:57.174123 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:06:57.174382 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:57.174386 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:57.175028 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:06:57.175029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:06:57.175391 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:57.175393 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:57.175665 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:57.176086 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:06:57.176099 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:06:57.177321 (+ 1222us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:06:57.178499 (+ 1178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:06:57.178503 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:06:57.178506 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:06:57.178698 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:06:57.178818 (+ 120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:06:57.178818 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:06:57.178819 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:06:57.178862 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:06:57.179006 (+ 144us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:06:57.179008 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:06:57.179096 (+ 88us) client_negotiation.cc:769] Sending connection context
0507 02:06:57.179175 (+ 79us) client_negotiation.cc:240] Negotiation successful
0507 02:06:57.179266 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":74,"threads_started":1}
I20240507 02:06:57.179833 2694 heartbeater.cc:347] Connected to a master server at 127.31.131.126:44827
I20240507 02:06:57.179908 2694 heartbeater.cc:464] Registering TS with master...
I20240507 02:06:57.180117 2694 heartbeater.cc:511] Master 127.31.131.126:44827 requested a full tablet report, sending...
I20240507 02:06:57.180444 2232 ts_manager.cc:194] Registered new tserver with Master: 3f1ef547e5394c1da3274eafad360a88 (127.31.131.67:45841)
I20240507 02:06:57.180737 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 2564
I20240507 02:06:57.180814 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-2/wal/instance
I20240507 02:06:57.180819 2232 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.131.67@KRBTEST.COM'} at 127.31.131.67:48361
I20240507 02:06:57.180984 32269 external_mini_cluster.cc:1428] Setting key 8d598e3e441c99b13cdb221932a33785
I20240507 02:06:57.181739 32269 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20240507 02:06:57.190076 2291 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:06:57.189323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36318 (local address 127.31.131.126:44827)
0507 02:06:57.189377 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:57.189379 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:57.189450 (+ 71us) server_negotiation.cc:407] Connection header received
0507 02:06:57.189515 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:57.189516 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:57.189525 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:57.189568 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:06:57.190022 (+ 454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36318: BlockingRecv error: recv got EOF from 127.0.0.1:36318 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
I20240507 02:06:57.190913 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 2294
I20240507 02:06:57.195132 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 2429
I20240507 02:06:57.199491 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 2564
I20240507 02:06:57.203337 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 2200
2024-05-07T02:06:57Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1594 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1715047604443657-32269-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 07 02:06:57 dist-test-slave-v0bg krb5kdc[2706](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 07 02:06:57 dist-test-slave-v0bg krb5kdc[2706](info): set up 2 sockets
May 07 02:06:57 dist-test-slave-v0bg krb5kdc[2706](info): commencing operation
krb5kdc: starting...
W20240507 02:06:58.079394 32269 mini_kdc.cc:121] Time spent starting KDC: real 0.858s user 0.003s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1715047604443657-32269-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1715047604443657-32269-0/krb5kdc/test-user.keytab.
May 07 02:06:58 dist-test-slave-v0bg krb5kdc[2706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-07T02:06:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-07T02:06:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.131.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab.
Entry for principal kudu/127.31.131.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab.
I20240507 02:06:58.179191 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1715047604443657-32269-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1715047604443657-32269-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1715047604443657-32269-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:36421
--webserver_interface=127.31.131.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab
--principal=kudu/127.31.131.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:39537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.131.126:36421
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:06:58.252643 2719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:06:58.252800 2719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:06:58.252820 2719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:06:58.254070 2719 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240507 02:06:58.254113 2719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:06:58.254127 2719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:06:58.254137 2719 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240507 02:06:58.254163 2719 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240507 02:06:58.255736 2719 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:39537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1715047604443657-32269-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1715047604443657-32269-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.131.126:36421
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab
--principal=kudu/127.31.131.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:36421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.131.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1715047604443657-32269-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:06:58.255928 2719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:06:58.256090 2719 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:06:58.259608 2727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:06:58.259742 2719 server_base.cc:1034] running on GCE node
W20240507 02:06:58.259621 2725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:06:58.259783 2724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:06:58.260077 2719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:06:58.260308 2719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:06:58.261428 2719 hybrid_clock.cc:648] HybridClock initialized: now 1715047618261410 us; error 35 us; skew 500 ppm
May 07 02:06:58 dist-test-slave-v0bg krb5kdc[2706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047618, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:06:58.263568 2719 init.cc:377] Logged in from keytab as kudu/127.31.131.126@KRBTEST.COM (short username kudu)
I20240507 02:06:58.263931 2719 webserver.cc:460] Webserver started at http://127.31.131.126:46655/ using document root <none> and password file <none>
I20240507 02:06:58.264075 2719 fs_manager.cc:359] Metadata directory not provided
I20240507 02:06:58.264117 2719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:06:58.264197 2719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:06:58.265130 2719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1715047604443657-32269-0/minicluster-data/master-0/data/instance:
uuid: "2a6390ae8af04ab29b306321c6a4006c"
format_stamp: "Formatted at 2024-05-07 02:06:58 on dist-test-slave-v0bg"
server_key: "19d9a3b1a3a6f9616bd9c2a32d7e47d9"
server_key_iv: "1677d601f4c2a78a18d7774f731dae0e"
server_key_version: "encryptionkey@0"
I20240507 02:06:58.265394 2719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1715047604443657-32269-0/minicluster-data/master-0/wal/instance:
uuid: "2a6390ae8af04ab29b306321c6a4006c"
format_stamp: "Formatted at 2024-05-07 02:06:58 on dist-test-slave-v0bg"
server_key: "19d9a3b1a3a6f9616bd9c2a32d7e47d9"
server_key_iv: "1677d601f4c2a78a18d7774f731dae0e"
server_key_version: "encryptionkey@0"
I20240507 02:06:58.266439 2719 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240507 02:06:58.267144 2734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:58.267298 2719 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240507 02:06:58.267365 2719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1715047604443657-32269-0/minicluster-data/master-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1715047604443657-32269-0/minicluster-data/master-0/wal
uuid: "2a6390ae8af04ab29b306321c6a4006c"
format_stamp: "Formatted at 2024-05-07 02:06:58 on dist-test-slave-v0bg"
server_key: "19d9a3b1a3a6f9616bd9c2a32d7e47d9"
server_key_iv: "1677d601f4c2a78a18d7774f731dae0e"
server_key_version: "encryptionkey@0"
I20240507 02:06:58.267410 2719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1715047604443657-32269-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1715047604443657-32269-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1715047604443657-32269-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:06:58.276046 2719 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:06:58.277483 2719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:06:58.277567 2719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:06:58.280542 2719 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.126:36421
I20240507 02:06:58.280577 2787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.126:36421 every 8 connection(s)
I20240507 02:06:58.280825 2719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
I20240507 02:06:58.281323 2788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:06:58.282232 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 2719
I20240507 02:06:58.282311 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1715047604443657-32269-0/minicluster-data/master-0/wal/instance
I20240507 02:06:58.282474 32269 external_mini_cluster.cc:1428] Setting key 33f3899b898cd34b41f3e88907546df3
I20240507 02:06:58.284188 2788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a6390ae8af04ab29b306321c6a4006c: Bootstrap starting.
I20240507 02:06:58.284736 2788 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a6390ae8af04ab29b306321c6a4006c: Neither blocks nor log segments found. Creating new log.
I20240507 02:06:58.285001 2788 log.cc:824] T 00000000000000000000000000000000 P 2a6390ae8af04ab29b306321c6a4006c: Log is configured to *not* fsync() on all Append() calls
I20240507 02:06:58.285607 2788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a6390ae8af04ab29b306321c6a4006c: No bootstrap required, opened a new log
I20240507 02:06:58.287287 2788 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a6390ae8af04ab29b306321c6a4006c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a6390ae8af04ab29b306321c6a4006c" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 36421 } }
I20240507 02:06:58.287428 2788 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a6390ae8af04ab29b306321c6a4006c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:06:58.287456 2788 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a6390ae8af04ab29b306321c6a4006c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a6390ae8af04ab29b306321c6a4006c, State: Initialized, Role: FOLLOWER
I20240507 02:06:58.287552 2788 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a6390ae8af04ab29b306321c6a4006c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a6390ae8af04ab29b306321c6a4006c" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 36421 } }
May 07 02:06:58 dist-test-slave-v0bg krb5kdc[2706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:06:58.287616 2788 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a6390ae8af04ab29b306321c6a4006c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 02:06:58.287649 2788 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a6390ae8af04ab29b306321c6a4006c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 02:06:58.287701 2788 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a6390ae8af04ab29b306321c6a4006c [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:06:58.288250 2788 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a6390ae8af04ab29b306321c6a4006c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a6390ae8af04ab29b306321c6a4006c" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 36421 } }
I20240507 02:06:58.288342 2788 leader_election.cc:302] T 00000000000000000000000000000000 P 2a6390ae8af04ab29b306321c6a4006c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a6390ae8af04ab29b306321c6a4006c; no voters:
I20240507 02:06:58.288482 2788 leader_election.cc:288] T 00000000000000000000000000000000 P 2a6390ae8af04ab29b306321c6a4006c [CANDIDATE]: Term 1 election: Requested vote from peers
I20240507 02:06:58.288532 2793 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a6390ae8af04ab29b306321c6a4006c [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:06:58.288697 2793 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a6390ae8af04ab29b306321c6a4006c [term 1 LEADER]: Becoming Leader. State: Replica: 2a6390ae8af04ab29b306321c6a4006c, State: Running, Role: LEADER
I20240507 02:06:58.288802 2788 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a6390ae8af04ab29b306321c6a4006c [sys.catalog]: configured and running, proceeding with master startup.
I20240507 02:06:58.288815 2793 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a6390ae8af04ab29b306321c6a4006c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a6390ae8af04ab29b306321c6a4006c" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 36421 } }
I20240507 02:06:58.289247 2793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a6390ae8af04ab29b306321c6a4006c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a6390ae8af04ab29b306321c6a4006c. Latest consensus state: current_term: 1 leader_uuid: "2a6390ae8af04ab29b306321c6a4006c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a6390ae8af04ab29b306321c6a4006c" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 36421 } } }
I20240507 02:06:58.289459 2793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a6390ae8af04ab29b306321c6a4006c [sys.catalog]: This master's current role is: LEADER
I20240507 02:06:58.289393 2795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a6390ae8af04ab29b306321c6a4006c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a6390ae8af04ab29b306321c6a4006c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a6390ae8af04ab29b306321c6a4006c" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 36421 } } }
I20240507 02:06:58.289798 2795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a6390ae8af04ab29b306321c6a4006c [sys.catalog]: This master's current role is: LEADER
I20240507 02:06:58.289920 2802 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 02:06:58.290375 2802 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240507 02:06:58.291630 2802 catalog_manager.cc:1347] Generated new cluster ID: b680d346891a48ddad920f41a0279e8c
I20240507 02:06:58.291674 2802 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240507 02:06:58.294953 2791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:58.282913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43902 (local address 127.31.131.126:36421)
0507 02:06:58.283052 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:58.283061 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:58.283090 (+ 29us) server_negotiation.cc:407] Connection header received
0507 02:06:58.283528 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:58.283536 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:58.283591 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:58.283744 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:06:58.284350 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:58.285364 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:58.285929 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:58.286158 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:58.288082 (+ 1924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:06:58.288089 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:06:58.288089 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:06:58.288102 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:06:58.290152 (+ 2050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:58.290495 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:58.290497 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:58.290498 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:58.290538 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:58.290858 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:58.290859 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:58.290860 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:58.291073 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:06:58.291218 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:58.294659 (+ 3441us) server_negotiation.cc:299] Negotiation successful
0507 02:06:58.294780 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":62,"threads_started":1}
I20240507 02:06:58.295858 32269 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240507 02:06:58.313979 2791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:58.310679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43906 (local address 127.31.131.126:36421)
0507 02:06:58.310729 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:58.310731 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:58.310934 (+ 203us) server_negotiation.cc:407] Connection header received
0507 02:06:58.310984 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:58.310985 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:58.310996 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:58.311060 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:06:58.311593 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:58.312312 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:58.312975 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:58.313154 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:58.313522 (+ 368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:06:58.313526 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:06:58.313526 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0507 02:06:58.313717 (+ 191us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:06:58.313734 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0507 02:06:58.313740 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:06:58.313798 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:58.313856 (+ 58us) server_negotiation.cc:299] Negotiation successful
0507 02:06:58.313892 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20240507 02:06:58.318459 2802 catalog_manager.cc:1370] Generated new certificate authority record
I20240507 02:06:58.318991 2802 catalog_manager.cc:1504] Loading token signing keys...
I20240507 02:06:58.325358 2802 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 2a6390ae8af04ab29b306321c6a4006c: Generated new TSK 0
I20240507 02:06:58.325510 2802 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240507 02:06:58.332942 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 2719
2024-05-07T02:06:58Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1128 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 07 02:06:58 dist-test-slave-v0bg krb5kdc[2821](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 07 02:06:58 dist-test-slave-v0bg krb5kdc[2821](info): set up 2 sockets
May 07 02:06:58 dist-test-slave-v0bg krb5kdc[2821](info): commencing operation
krb5kdc: starting...
W20240507 02:06:59.219463 32269 mini_kdc.cc:121] Time spent starting KDC: real 0.870s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/krb5kdc/test-user.keytab.
May 07 02:06:59 dist-test-slave-v0bg krb5kdc[2821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047619, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-07T02:06:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-07T02:06:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.131.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab.
Entry for principal kudu/127.31.131.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab.
I20240507 02:06:59.319046 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:40387
--webserver_interface=127.31.131.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab
--principal=kudu/127.31.131.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:44259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.131.126:40387
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:06:59.394024 2834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:06:59.394222 2834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:06:59.394251 2834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:06:59.395535 2834 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240507 02:06:59.395578 2834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:06:59.395596 2834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:06:59.395612 2834 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240507 02:06:59.395625 2834 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240507 02:06:59.397068 2834 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:44259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.131.126:40387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab
--principal=kudu/127.31.131.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:40387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.131.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:06:59.397279 2834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:06:59.397452 2834 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:06:59.401052 2842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:06:59.401070 2839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:06:59.401296 2834 server_base.cc:1034] running on GCE node
W20240507 02:06:59.401368 2840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:06:59.401633 2834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:06:59.401890 2834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:06:59.403020 2834 hybrid_clock.cc:648] HybridClock initialized: now 1715047619403019 us; error 37 us; skew 500 ppm
May 07 02:06:59 dist-test-slave-v0bg krb5kdc[2821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047619, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:06:59.405145 2834 init.cc:377] Logged in from keytab as kudu/127.31.131.126@KRBTEST.COM (short username kudu)
I20240507 02:06:59.405501 2834 webserver.cc:460] Webserver started at http://127.31.131.126:40027/ using document root <none> and password file <none>
I20240507 02:06:59.405651 2834 fs_manager.cc:359] Metadata directory not provided
I20240507 02:06:59.405690 2834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:06:59.405802 2834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:06:59.406750 2834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/minicluster-data/master-0/data/instance:
uuid: "da7756d3c8b74c97b92c0894b71d5884"
format_stamp: "Formatted at 2024-05-07 02:06:59 on dist-test-slave-v0bg"
server_key: "057d863694116820eb379ca8cd943ab7"
server_key_iv: "cf1e034b920ec73a6fd5c411a6ba5e49"
server_key_version: "encryptionkey@0"
I20240507 02:06:59.407029 2834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/minicluster-data/master-0/wal/instance:
uuid: "da7756d3c8b74c97b92c0894b71d5884"
format_stamp: "Formatted at 2024-05-07 02:06:59 on dist-test-slave-v0bg"
server_key: "057d863694116820eb379ca8cd943ab7"
server_key_iv: "cf1e034b920ec73a6fd5c411a6ba5e49"
server_key_version: "encryptionkey@0"
I20240507 02:06:59.408057 2834 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240507 02:06:59.408630 2849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:59.408783 2834 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:59.408850 2834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/minicluster-data/master-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/minicluster-data/master-0/wal
uuid: "da7756d3c8b74c97b92c0894b71d5884"
format_stamp: "Formatted at 2024-05-07 02:06:59 on dist-test-slave-v0bg"
server_key: "057d863694116820eb379ca8cd943ab7"
server_key_iv: "cf1e034b920ec73a6fd5c411a6ba5e49"
server_key_version: "encryptionkey@0"
I20240507 02:06:59.408906 2834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:06:59.420068 2834 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:06:59.421590 2834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:06:59.421703 2834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:06:59.425096 2834 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.126:40387
I20240507 02:06:59.425143 2902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.126:40387 every 8 connection(s)
I20240507 02:06:59.425421 2834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
I20240507 02:06:59.426229 2903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:06:59.428263 2903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da7756d3c8b74c97b92c0894b71d5884: Bootstrap starting.
I20240507 02:06:59.428754 2903 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da7756d3c8b74c97b92c0894b71d5884: Neither blocks nor log segments found. Creating new log.
I20240507 02:06:59.428951 2903 log.cc:824] T 00000000000000000000000000000000 P da7756d3c8b74c97b92c0894b71d5884: Log is configured to *not* fsync() on all Append() calls
I20240507 02:06:59.429498 2903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da7756d3c8b74c97b92c0894b71d5884: No bootstrap required, opened a new log
I20240507 02:06:59.430883 2903 raft_consensus.cc:348] T 00000000000000000000000000000000 P da7756d3c8b74c97b92c0894b71d5884 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da7756d3c8b74c97b92c0894b71d5884" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 40387 } }
I20240507 02:06:59.430999 2903 raft_consensus.cc:374] T 00000000000000000000000000000000 P da7756d3c8b74c97b92c0894b71d5884 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:06:59.431015 2903 raft_consensus.cc:729] T 00000000000000000000000000000000 P da7756d3c8b74c97b92c0894b71d5884 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da7756d3c8b74c97b92c0894b71d5884, State: Initialized, Role: FOLLOWER
I20240507 02:06:59.431114 2903 consensus_queue.cc:260] T 00000000000000000000000000000000 P da7756d3c8b74c97b92c0894b71d5884 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da7756d3c8b74c97b92c0894b71d5884" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 40387 } }
I20240507 02:06:59.431182 2903 raft_consensus.cc:388] T 00000000000000000000000000000000 P da7756d3c8b74c97b92c0894b71d5884 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 02:06:59.431208 2903 raft_consensus.cc:482] T 00000000000000000000000000000000 P da7756d3c8b74c97b92c0894b71d5884 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 02:06:59.431236 2903 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da7756d3c8b74c97b92c0894b71d5884 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:06:59.431681 2903 raft_consensus.cc:504] T 00000000000000000000000000000000 P da7756d3c8b74c97b92c0894b71d5884 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da7756d3c8b74c97b92c0894b71d5884" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 40387 } }
I20240507 02:06:59.431766 2903 leader_election.cc:302] T 00000000000000000000000000000000 P da7756d3c8b74c97b92c0894b71d5884 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da7756d3c8b74c97b92c0894b71d5884; no voters:
I20240507 02:06:59.431880 2903 leader_election.cc:288] T 00000000000000000000000000000000 P da7756d3c8b74c97b92c0894b71d5884 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240507 02:06:59.432062 2906 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da7756d3c8b74c97b92c0894b71d5884 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:06:59.432077 2903 sys_catalog.cc:564] T 00000000000000000000000000000000 P da7756d3c8b74c97b92c0894b71d5884 [sys.catalog]: configured and running, proceeding with master startup.
I20240507 02:06:59.432226 2906 raft_consensus.cc:686] T 00000000000000000000000000000000 P da7756d3c8b74c97b92c0894b71d5884 [term 1 LEADER]: Becoming Leader. State: Replica: da7756d3c8b74c97b92c0894b71d5884, State: Running, Role: LEADER
I20240507 02:06:59.432341 2906 consensus_queue.cc:237] T 00000000000000000000000000000000 P da7756d3c8b74c97b92c0894b71d5884 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da7756d3c8b74c97b92c0894b71d5884" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 40387 } }
I20240507 02:06:59.432487 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 2834
I20240507 02:06:59.432587 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/minicluster-data/master-0/wal/instance
I20240507 02:06:59.432687 2907 sys_catalog.cc:455] T 00000000000000000000000000000000 P da7756d3c8b74c97b92c0894b71d5884 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da7756d3c8b74c97b92c0894b71d5884" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da7756d3c8b74c97b92c0894b71d5884" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 40387 } } }
I20240507 02:06:59.432754 32269 external_mini_cluster.cc:1428] Setting key 2f57ac1cbe3b420ac11db682e7be109d
I20240507 02:06:59.432749 2907 sys_catalog.cc:458] T 00000000000000000000000000000000 P da7756d3c8b74c97b92c0894b71d5884 [sys.catalog]: This master's current role is: LEADER
I20240507 02:06:59.432889 2908 sys_catalog.cc:455] T 00000000000000000000000000000000 P da7756d3c8b74c97b92c0894b71d5884 [sys.catalog]: SysCatalogTable state changed. Reason: New leader da7756d3c8b74c97b92c0894b71d5884. Latest consensus state: current_term: 1 leader_uuid: "da7756d3c8b74c97b92c0894b71d5884" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da7756d3c8b74c97b92c0894b71d5884" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 40387 } } }
I20240507 02:06:59.433259 2908 sys_catalog.cc:458] T 00000000000000000000000000000000 P da7756d3c8b74c97b92c0894b71d5884 [sys.catalog]: This master's current role is: LEADER
W20240507 02:06:59.433944 2923 catalog_manager.cc:1558] T 00000000000000000000000000000000 P da7756d3c8b74c97b92c0894b71d5884: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240507 02:06:59.433993 2923 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240507 02:06:59.434041 2915 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 02:06:59.434231 2915 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240507 02:06:59.436395 2915 catalog_manager.cc:1347] Generated new cluster ID: e3ceb8b546424e69941cd030feb3a7af
I20240507 02:06:59.436450 2915 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
May 07 02:06:59 dist-test-slave-v0bg krb5kdc[2821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047619, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:06:59.441308 2920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:59.433181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53046 (local address 127.31.131.126:40387)
0507 02:06:59.433411 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:59.433438 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:59.433664 (+ 226us) server_negotiation.cc:407] Connection header received
0507 02:06:59.433831 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:59.433838 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:59.433889 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:59.434039 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:06:59.434851 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:59.435990 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:59.436565 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:59.436786 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:59.438596 (+ 1810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:06:59.438619 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:06:59.438619 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:06:59.438634 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:06:59.440109 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:59.440392 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:59.440394 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:59.440395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:59.440441 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:59.440561 (+ 120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:59.440562 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:59.440562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:59.440727 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:06:59.440853 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:59.440965 (+ 112us) server_negotiation.cc:299] Negotiation successful
0507 02:06:59.441075 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":73,"threads_started":1}
I20240507 02:06:59.461953 2915 catalog_manager.cc:1370] Generated new certificate authority record
I20240507 02:06:59.462499 2915 catalog_manager.cc:1504] Loading token signing keys...
I20240507 02:06:59.467437 2915 catalog_manager.cc:5857] T 00000000000000000000000000000000 P da7756d3c8b74c97b92c0894b71d5884: Generated new TSK 0
I20240507 02:06:59.467638 2915 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.131.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab.
Entry for principal kudu/127.31.131.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab.
I20240507 02:06:59.496574 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.65:0
--local_ip_for_outbound_sockets=127.31.131.65
--webserver_interface=127.31.131.65
--webserver_port=0
--tserver_master_addrs=127.31.131.126:40387
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab
--principal=kudu/127.31.131.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:44259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:06:59.586468 2929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:06:59.586686 2929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:06:59.586717 2929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:06:59.587985 2929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:06:59.588022 2929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:06:59.588053 2929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.65
I20240507 02:06:59.589469 2929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:44259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab
--principal=kudu/127.31.131.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.131.65
--webserver_port=0
--tserver_master_addrs=127.31.131.126:40387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.65
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:06:59.589664 2929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:06:59.589834 2929 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:06:59.593520 2937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:06:59.593569 2934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:06:59.593631 2935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:06:59.593638 2929 server_base.cc:1034] running on GCE node
I20240507 02:06:59.593919 2929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:06:59.594112 2929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:06:59.595244 2929 hybrid_clock.cc:648] HybridClock initialized: now 1715047619595229 us; error 38 us; skew 500 ppm
May 07 02:06:59 dist-test-slave-v0bg krb5kdc[2821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047619, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:06:59.597659 2929 init.cc:377] Logged in from keytab as kudu/127.31.131.65@KRBTEST.COM (short username kudu)
I20240507 02:06:59.598098 2929 webserver.cc:460] Webserver started at http://127.31.131.65:33305/ using document root <none> and password file <none>
I20240507 02:06:59.598299 2929 fs_manager.cc:359] Metadata directory not provided
I20240507 02:06:59.598346 2929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:06:59.598448 2929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:06:59.599623 2929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-0/data/instance:
uuid: "ee11dc1676c74b949e7432d913eccb49"
format_stamp: "Formatted at 2024-05-07 02:06:59 on dist-test-slave-v0bg"
server_key: "c2fb473a14bcc61601d180185ed3ddc4"
server_key_iv: "0689e2169f39c626edbc21062c0eb7e8"
server_key_version: "encryptionkey@0"
I20240507 02:06:59.599956 2929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-0/wal/instance:
uuid: "ee11dc1676c74b949e7432d913eccb49"
format_stamp: "Formatted at 2024-05-07 02:06:59 on dist-test-slave-v0bg"
server_key: "c2fb473a14bcc61601d180185ed3ddc4"
server_key_iv: "0689e2169f39c626edbc21062c0eb7e8"
server_key_version: "encryptionkey@0"
I20240507 02:06:59.601361 2929 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240507 02:06:59.602053 2944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:59.602201 2929 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240507 02:06:59.602281 2929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-0/wal
uuid: "ee11dc1676c74b949e7432d913eccb49"
format_stamp: "Formatted at 2024-05-07 02:06:59 on dist-test-slave-v0bg"
server_key: "c2fb473a14bcc61601d180185ed3ddc4"
server_key_iv: "0689e2169f39c626edbc21062c0eb7e8"
server_key_version: "encryptionkey@0"
I20240507 02:06:59.602339 2929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:06:59.611558 2929 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:06:59.612946 2929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:06:59.613025 2929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:06:59.613157 2929 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:06:59.613373 2929 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:06:59.613400 2929 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:59.613428 2929 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:06:59.613445 2929 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:59.618194 2929 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.65:43551
I20240507 02:06:59.618240 3058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.65:43551 every 8 connection(s)
I20240507 02:06:59.618459 2929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
I20240507 02:06:59.620784 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 2929
I20240507 02:06:59.620878 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-0/wal/instance
I20240507 02:06:59.621021 32269 external_mini_cluster.cc:1428] Setting key e8d16d103e96ec3c2bfbaa3274f9f7ee
May 07 02:06:59 dist-test-slave-v0bg krb5kdc[2821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047619, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.65@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:06:59.626230 3061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:59.619078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40387 (local address 127.31.131.65:57247)
0507 02:06:59.619198 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0507 02:06:59.619221 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0507 02:06:59.619604 (+ 383us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:06:59.619791 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:06:59.619794 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:06:59.620031 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:06:59.620522 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:59.620526 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:59.621380 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:06:59.621381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:06:59.621789 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:59.621791 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:59.621881 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:59.622294 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:06:59.622307 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:06:59.623690 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:06:59.625294 (+ 1604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:06:59.625299 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:06:59.625301 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:06:59.625473 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:06:59.625620 (+ 147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:06:59.625621 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:06:59.625621 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:06:59.625661 (+ 40us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:06:59.625867 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:06:59.625870 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:06:59.625954 (+ 84us) client_negotiation.cc:769] Sending connection context
0507 02:06:59.626026 (+ 72us) client_negotiation.cc:240] Negotiation successful
0507 02:06:59.626109 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":60,"threads_started":1}
I20240507 02:06:59.626243 2920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:59.619099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:57247 (local address 127.31.131.126:40387)
0507 02:06:59.619161 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:59.619164 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:59.619286 (+ 122us) server_negotiation.cc:407] Connection header received
0507 02:06:59.619674 (+ 388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:59.619675 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:59.619700 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:59.619746 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:06:59.620633 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:59.621293 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:59.621859 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:59.622025 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:59.623773 (+ 1748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:06:59.623777 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:06:59.623777 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:06:59.623784 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:06:59.625220 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:59.625537 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:59.625539 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:59.625539 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:59.625565 (+ 26us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:59.625708 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:59.625709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:59.625710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:59.625815 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:06:59.625883 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:59.626060 (+ 177us) server_negotiation.cc:299] Negotiation successful
0507 02:06:59.626145 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20240507 02:06:59.626814 3059 heartbeater.cc:347] Connected to a master server at 127.31.131.126:40387
I20240507 02:06:59.626896 3059 heartbeater.cc:464] Registering TS with master...
I20240507 02:06:59.627058 3059 heartbeater.cc:511] Master 127.31.131.126:40387 requested a full tablet report, sending...
I20240507 02:06:59.627528 2866 ts_manager.cc:194] Registered new tserver with Master: ee11dc1676c74b949e7432d913eccb49 (127.31.131.65:43551)
WARNING: no policy specified for kudu/127.31.131.66@KRBTEST.COM; defaulting to no policy
I20240507 02:06:59.628451 2866 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.131.65@KRBTEST.COM'} at 127.31.131.65:57247
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab.
Entry for principal kudu/127.31.131.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab.
I20240507 02:06:59.642159 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.66:0
--local_ip_for_outbound_sockets=127.31.131.66
--webserver_interface=127.31.131.66
--webserver_port=0
--tserver_master_addrs=127.31.131.126:40387
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab
--principal=kudu/127.31.131.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:44259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:06:59.716926 3064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:06:59.717108 3064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:06:59.717128 3064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:06:59.718328 3064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:06:59.718364 3064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:06:59.718385 3064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.66
I20240507 02:06:59.719856 3064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:44259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab
--principal=kudu/127.31.131.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.131.66
--webserver_port=0
--tserver_master_addrs=127.31.131.126:40387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.66
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:06:59.720029 3064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:06:59.720175 3064 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:06:59.724146 3072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:06:59.724135 3070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:06:59.724134 3069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:06:59.725061 3064 server_base.cc:1034] running on GCE node
I20240507 02:06:59.725222 3064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:06:59.725414 3064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:06:59.726541 3064 hybrid_clock.cc:648] HybridClock initialized: now 1715047619726528 us; error 32 us; skew 500 ppm
May 07 02:06:59 dist-test-slave-v0bg krb5kdc[2821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047619, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:06:59.728742 3064 init.cc:377] Logged in from keytab as kudu/127.31.131.66@KRBTEST.COM (short username kudu)
I20240507 02:06:59.729120 3064 webserver.cc:460] Webserver started at http://127.31.131.66:36927/ using document root <none> and password file <none>
I20240507 02:06:59.729280 3064 fs_manager.cc:359] Metadata directory not provided
I20240507 02:06:59.729318 3064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:06:59.729418 3064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:06:59.730332 3064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-1/data/instance:
uuid: "daa0795f52fe45edb7d45a59e747887f"
format_stamp: "Formatted at 2024-05-07 02:06:59 on dist-test-slave-v0bg"
server_key: "abeec23eecf7ae989d58d0543fbbfd21"
server_key_iv: "3cfbbce58ea8c0683cad7e8f106e62a0"
server_key_version: "encryptionkey@0"
I20240507 02:06:59.730641 3064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-1/wal/instance:
uuid: "daa0795f52fe45edb7d45a59e747887f"
format_stamp: "Formatted at 2024-05-07 02:06:59 on dist-test-slave-v0bg"
server_key: "abeec23eecf7ae989d58d0543fbbfd21"
server_key_iv: "3cfbbce58ea8c0683cad7e8f106e62a0"
server_key_version: "encryptionkey@0"
I20240507 02:06:59.731683 3064 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240507 02:06:59.732295 3079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:59.732431 3064 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240507 02:06:59.732504 3064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-1/wal
uuid: "daa0795f52fe45edb7d45a59e747887f"
format_stamp: "Formatted at 2024-05-07 02:06:59 on dist-test-slave-v0bg"
server_key: "abeec23eecf7ae989d58d0543fbbfd21"
server_key_iv: "3cfbbce58ea8c0683cad7e8f106e62a0"
server_key_version: "encryptionkey@0"
I20240507 02:06:59.732559 3064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:06:59.740888 3064 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:06:59.742270 3064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:06:59.742372 3064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:06:59.742524 3064 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:06:59.742812 3064 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:06:59.742843 3064 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:59.742877 3064 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:06:59.742902 3064 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:59.747731 3064 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.66:41881
I20240507 02:06:59.747790 3192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.66:41881 every 8 connection(s)
I20240507 02:06:59.748031 3064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
May 07 02:06:59 dist-test-slave-v0bg krb5kdc[2821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047619, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.66@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:06:59.755004 2920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:59.748625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.66:39065 (local address 127.31.131.126:40387)
0507 02:06:59.748674 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:59.748676 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:59.748822 (+ 146us) server_negotiation.cc:407] Connection header received
0507 02:06:59.749284 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:59.749285 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:59.749294 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:59.749328 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:06:59.750225 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:59.750741 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:59.751230 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:59.751331 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:59.752968 (+ 1637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:06:59.752972 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:06:59.752972 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:06:59.752978 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:06:59.754151 (+ 1173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:59.754409 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:59.754410 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:59.754410 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:59.754424 (+ 14us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:59.754536 (+ 112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:59.754537 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:59.754537 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:59.754644 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:06:59.754691 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:59.754878 (+ 187us) server_negotiation.cc:299] Negotiation successful
0507 02:06:59.754939 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20240507 02:06:59.755115 3195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:59.748599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40387 (local address 127.31.131.66:39065)
0507 02:06:59.748724 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0507 02:06:59.748749 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0507 02:06:59.749191 (+ 442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:06:59.749383 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:06:59.749386 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:06:59.749643 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:06:59.750153 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:59.750157 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:59.750811 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:06:59.750812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:06:59.751175 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:59.751176 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:59.751273 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:59.751677 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:06:59.751689 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:06:59.752887 (+ 1198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:06:59.754209 (+ 1322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:06:59.754212 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:06:59.754214 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:06:59.754362 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:06:59.754462 (+ 100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:06:59.754463 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:06:59.754463 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:06:59.754504 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:06:59.754688 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:06:59.754692 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:06:59.754788 (+ 96us) client_negotiation.cc:769] Sending connection context
0507 02:06:59.754874 (+ 86us) client_negotiation.cc:240] Negotiation successful
0507 02:06:59.754966 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":67,"threads_started":1}
I20240507 02:06:59.755206 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 3064
I20240507 02:06:59.755288 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-1/wal/instance
I20240507 02:06:59.755434 32269 external_mini_cluster.cc:1428] Setting key 81c4e814c6dd84b2b772fa7e1591d70b
I20240507 02:06:59.755586 3193 heartbeater.cc:347] Connected to a master server at 127.31.131.126:40387
I20240507 02:06:59.755671 3193 heartbeater.cc:464] Registering TS with master...
I20240507 02:06:59.755885 3193 heartbeater.cc:511] Master 127.31.131.126:40387 requested a full tablet report, sending...
I20240507 02:06:59.756246 2866 ts_manager.cc:194] Registered new tserver with Master: daa0795f52fe45edb7d45a59e747887f (127.31.131.66:41881)
I20240507 02:06:59.756611 2866 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.131.66@KRBTEST.COM'} at 127.31.131.66:39065
WARNING: no policy specified for kudu/127.31.131.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab.
Entry for principal kudu/127.31.131.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab.
I20240507 02:06:59.774880 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.67:0
--local_ip_for_outbound_sockets=127.31.131.67
--webserver_interface=127.31.131.67
--webserver_port=0
--tserver_master_addrs=127.31.131.126:40387
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab
--principal=kudu/127.31.131.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:44259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:06:59.850189 3198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:06:59.850370 3198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:06:59.850390 3198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:06:59.851651 3198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:06:59.851692 3198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:06:59.851716 3198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.67
I20240507 02:06:59.853133 3198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:44259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab
--principal=kudu/127.31.131.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.131.67
--webserver_port=0
--tserver_master_addrs=127.31.131.126:40387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.67
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:06:59.853303 3198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:06:59.853473 3198 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:06:59.857275 3203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:06:59.857275 3206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:06:59.857422 3198 server_base.cc:1034] running on GCE node
W20240507 02:06:59.857424 3204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:06:59.857620 3198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:06:59.857822 3198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:06:59.858950 3198 hybrid_clock.cc:648] HybridClock initialized: now 1715047619858934 us; error 32 us; skew 500 ppm
May 07 02:06:59 dist-test-slave-v0bg krb5kdc[2821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047619, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:06:59.861088 3198 init.cc:377] Logged in from keytab as kudu/127.31.131.67@KRBTEST.COM (short username kudu)
I20240507 02:06:59.861423 3198 webserver.cc:460] Webserver started at http://127.31.131.67:36571/ using document root <none> and password file <none>
I20240507 02:06:59.861557 3198 fs_manager.cc:359] Metadata directory not provided
I20240507 02:06:59.861589 3198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:06:59.861662 3198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:06:59.862577 3198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-2/data/instance:
uuid: "24813f68af01428fbc2962065474db02"
format_stamp: "Formatted at 2024-05-07 02:06:59 on dist-test-slave-v0bg"
server_key: "57aee337301e0a891fd2d30e977a3cb5"
server_key_iv: "ade1d85dfb6cf1b0881e03a21b1719e9"
server_key_version: "encryptionkey@0"
I20240507 02:06:59.862839 3198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-2/wal/instance:
uuid: "24813f68af01428fbc2962065474db02"
format_stamp: "Formatted at 2024-05-07 02:06:59 on dist-test-slave-v0bg"
server_key: "57aee337301e0a891fd2d30e977a3cb5"
server_key_iv: "ade1d85dfb6cf1b0881e03a21b1719e9"
server_key_version: "encryptionkey@0"
I20240507 02:06:59.863830 3198 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240507 02:06:59.864379 3213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:59.864531 3198 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240507 02:06:59.864596 3198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-2/wal
uuid: "24813f68af01428fbc2962065474db02"
format_stamp: "Formatted at 2024-05-07 02:06:59 on dist-test-slave-v0bg"
server_key: "57aee337301e0a891fd2d30e977a3cb5"
server_key_iv: "ade1d85dfb6cf1b0881e03a21b1719e9"
server_key_version: "encryptionkey@0"
I20240507 02:06:59.864643 3198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:06:59.887396 3198 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:06:59.888944 3198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:06:59.889035 3198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:06:59.889202 3198 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:06:59.889489 3198 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:06:59.889520 3198 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:59.889541 3198 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:06:59.889565 3198 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:06:59.894325 3198 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.67:36033
I20240507 02:06:59.894382 3328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.67:36033 every 8 connection(s)
I20240507 02:06:59.894663 3198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
I20240507 02:06:59.898383 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 3198
I20240507 02:06:59.898460 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1715047604443657-32269-0/minicluster-data/ts-2/wal/instance
I20240507 02:06:59.898641 32269 external_mini_cluster.cc:1428] Setting key 7d84c91d1a3420a335f8f924bd50169f
May 07 02:06:59 dist-test-slave-v0bg krb5kdc[2821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047619, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.67@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:06:59.901706 2920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:59.895284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.67:42445 (local address 127.31.131.126:40387)
0507 02:06:59.895336 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:59.895337 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:59.895481 (+ 144us) server_negotiation.cc:407] Connection header received
0507 02:06:59.895911 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:59.895912 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:59.895919 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:59.895974 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:06:59.896874 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:59.897343 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:59.897802 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:59.897900 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:59.899731 (+ 1831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:06:59.899733 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:06:59.899734 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:06:59.899739 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:06:59.900918 (+ 1179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:59.901179 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:59.901179 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:59.901180 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:59.901195 (+ 15us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:59.901306 (+ 111us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:59.901306 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:59.901306 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:59.901380 (+ 74us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:06:59.901419 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:59.901592 (+ 173us) server_negotiation.cc:299] Negotiation successful
0507 02:06:59.901646 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20240507 02:06:59.901795 3331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:59.895246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40387 (local address 127.31.131.67:42445)
0507 02:06:59.895376 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0507 02:06:59.895404 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0507 02:06:59.895828 (+ 424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:06:59.896027 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:06:59.896031 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:06:59.896299 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:06:59.896795 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:59.896798 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:59.897397 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:06:59.897398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:06:59.897751 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:06:59.897752 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:06:59.897849 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:59.898259 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:06:59.898271 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:06:59.899659 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:06:59.900974 (+ 1315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:06:59.900977 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:06:59.900979 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:06:59.901133 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:06:59.901234 (+ 101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:06:59.901234 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:06:59.901235 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:06:59.901276 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:06:59.901416 (+ 140us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:06:59.901418 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:06:59.901507 (+ 89us) client_negotiation.cc:769] Sending connection context
0507 02:06:59.901580 (+ 73us) client_negotiation.cc:240] Negotiation successful
0507 02:06:59.901670 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":53,"threads_started":1}
I20240507 02:06:59.902161 3329 heartbeater.cc:347] Connected to a master server at 127.31.131.126:40387
I20240507 02:06:59.902228 3329 heartbeater.cc:464] Registering TS with master...
I20240507 02:06:59.902402 3329 heartbeater.cc:511] Master 127.31.131.126:40387 requested a full tablet report, sending...
I20240507 02:06:59.902731 2866 ts_manager.cc:194] Registered new tserver with Master: 24813f68af01428fbc2962065474db02 (127.31.131.67:36033)
I20240507 02:06:59.903086 2866 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.131.67@KRBTEST.COM'} at 127.31.131.67:42445
I20240507 02:06:59.910166 32269 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
May 07 02:06:59 dist-test-slave-v0bg krb5kdc[2821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047619, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
May 07 02:06:59 dist-test-slave-v0bg krb5kdc[2821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047619, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:06:59.928529 2920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:06:59.922914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53048 (local address 127.31.131.126:40387)
0507 02:06:59.922970 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0507 02:06:59.922972 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:06:59.923048 (+ 76us) server_negotiation.cc:407] Connection header received
0507 02:06:59.923118 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:06:59.923118 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:06:59.923128 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:06:59.923171 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:06:59.923833 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:59.924299 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:06:59.924836 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:06:59.924994 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:06:59.926710 (+ 1716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:06:59.926714 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:06:59.926714 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:06:59.926720 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:06:59.927708 (+ 988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:59.927961 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:59.927962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:59.927962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:59.927981 (+ 19us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:06:59.928094 (+ 113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:06:59.928095 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:06:59.928095 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:06:59.928197 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:06:59.928289 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0507 02:06:59.928367 (+ 78us) server_negotiation.cc:299] Negotiation successful
0507 02:06:59.928436 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
W20240507 02:06:59.928725 2866 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:53048
I20240507 02:06:59.929047 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 2929
I20240507 02:06:59.932935 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 3064
I20240507 02:06:59.936353 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 3198
I20240507 02:06:59.939538 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 2834
2024-05-07T02:06:59Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1607 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 07 02:06:59 dist-test-slave-v0bg krb5kdc[3341](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 07 02:06:59 dist-test-slave-v0bg krb5kdc[3341](info): set up 2 sockets
May 07 02:06:59 dist-test-slave-v0bg krb5kdc[3341](info): commencing operation
krb5kdc: starting...
W20240507 02:07:00.827526 32269 mini_kdc.cc:121] Time spent starting KDC: real 0.870s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/krb5kdc/test-user.keytab.
May 07 02:07:00 dist-test-slave-v0bg krb5kdc[3341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-07T02:07:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-07T02:07:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.131.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab.
Entry for principal kudu/127.31.131.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab.
I20240507 02:07:00.929720 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:45979
--webserver_interface=127.31.131.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab
--principal=kudu/127.31.131.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:43217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.131.126:45979
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:07:01.004244 3354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:07:01.004405 3354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:07:01.004424 3354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:07:01.005664 3354 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240507 02:07:01.005718 3354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:07:01.005733 3354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:07:01.005743 3354 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240507 02:07:01.005753 3354 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240507 02:07:01.007321 3354 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:43217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.131.126:45979
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab
--principal=kudu/127.31.131.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:45979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.131.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3354
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:07:01.007513 3354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:07:01.007683 3354 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:07:01.011162 3360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:07:01.011157 3359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:07:01.011281 3354 server_base.cc:1034] running on GCE node
W20240507 02:07:01.011371 3362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:07:01.011574 3354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:07:01.011809 3354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:07:01.012936 3354 hybrid_clock.cc:648] HybridClock initialized: now 1715047621012917 us; error 37 us; skew 500 ppm
May 07 02:07:01 dist-test-slave-v0bg krb5kdc[3341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047621, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:07:01.019779 3354 init.cc:377] Logged in from keytab as kudu/127.31.131.126@KRBTEST.COM (short username kudu)
I20240507 02:07:01.020188 3354 webserver.cc:460] Webserver started at http://127.31.131.126:34895/ using document root <none> and password file <none>
I20240507 02:07:01.020364 3354 fs_manager.cc:359] Metadata directory not provided
I20240507 02:07:01.020406 3354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:07:01.020506 3354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:07:01.021441 3354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/minicluster-data/master-0/data/instance:
uuid: "b97a91efb55a4cd9904e0476fddacc09"
format_stamp: "Formatted at 2024-05-07 02:07:01 on dist-test-slave-v0bg"
server_key: "b06577c67ce0ad788d50fb5c2b90e7a9"
server_key_iv: "d6aae326076999688325982d3685f44b"
server_key_version: "encryptionkey@0"
I20240507 02:07:01.021765 3354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/minicluster-data/master-0/wal/instance:
uuid: "b97a91efb55a4cd9904e0476fddacc09"
format_stamp: "Formatted at 2024-05-07 02:07:01 on dist-test-slave-v0bg"
server_key: "b06577c67ce0ad788d50fb5c2b90e7a9"
server_key_iv: "d6aae326076999688325982d3685f44b"
server_key_version: "encryptionkey@0"
I20240507 02:07:01.022886 3354 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240507 02:07:01.023478 3369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:01.023613 3354 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240507 02:07:01.023684 3354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/minicluster-data/master-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/minicluster-data/master-0/wal
uuid: "b97a91efb55a4cd9904e0476fddacc09"
format_stamp: "Formatted at 2024-05-07 02:07:01 on dist-test-slave-v0bg"
server_key: "b06577c67ce0ad788d50fb5c2b90e7a9"
server_key_iv: "d6aae326076999688325982d3685f44b"
server_key_version: "encryptionkey@0"
I20240507 02:07:01.023743 3354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:07:01.054006 3354 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:07:01.055724 3354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:07:01.055855 3354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:07:01.059119 3354 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.126:45979
I20240507 02:07:01.059155 3421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.126:45979 every 8 connection(s)
I20240507 02:07:01.059438 3354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
I20240507 02:07:01.059967 3422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:07:01.062198 3422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b97a91efb55a4cd9904e0476fddacc09: Bootstrap starting.
I20240507 02:07:01.062716 3422 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b97a91efb55a4cd9904e0476fddacc09: Neither blocks nor log segments found. Creating new log.
I20240507 02:07:01.062911 3422 log.cc:824] T 00000000000000000000000000000000 P b97a91efb55a4cd9904e0476fddacc09: Log is configured to *not* fsync() on all Append() calls
I20240507 02:07:01.063465 3422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b97a91efb55a4cd9904e0476fddacc09: No bootstrap required, opened a new log
I20240507 02:07:01.063607 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 3354
I20240507 02:07:01.063701 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/minicluster-data/master-0/wal/instance
I20240507 02:07:01.063880 32269 external_mini_cluster.cc:1428] Setting key 9a4f5dec56ca8752a77ad17601bacd83
I20240507 02:07:01.065125 3422 raft_consensus.cc:348] T 00000000000000000000000000000000 P b97a91efb55a4cd9904e0476fddacc09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b97a91efb55a4cd9904e0476fddacc09" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 45979 } }
I20240507 02:07:01.065254 3422 raft_consensus.cc:374] T 00000000000000000000000000000000 P b97a91efb55a4cd9904e0476fddacc09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:07:01.065279 3422 raft_consensus.cc:729] T 00000000000000000000000000000000 P b97a91efb55a4cd9904e0476fddacc09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b97a91efb55a4cd9904e0476fddacc09, State: Initialized, Role: FOLLOWER
I20240507 02:07:01.065412 3422 consensus_queue.cc:260] T 00000000000000000000000000000000 P b97a91efb55a4cd9904e0476fddacc09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b97a91efb55a4cd9904e0476fddacc09" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 45979 } }
I20240507 02:07:01.065485 3422 raft_consensus.cc:388] T 00000000000000000000000000000000 P b97a91efb55a4cd9904e0476fddacc09 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 02:07:01.065515 3422 raft_consensus.cc:482] T 00000000000000000000000000000000 P b97a91efb55a4cd9904e0476fddacc09 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 02:07:01.065558 3422 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b97a91efb55a4cd9904e0476fddacc09 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:07:01.066164 3422 raft_consensus.cc:504] T 00000000000000000000000000000000 P b97a91efb55a4cd9904e0476fddacc09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b97a91efb55a4cd9904e0476fddacc09" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 45979 } }
I20240507 02:07:01.066256 3422 leader_election.cc:302] T 00000000000000000000000000000000 P b97a91efb55a4cd9904e0476fddacc09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b97a91efb55a4cd9904e0476fddacc09; no voters:
I20240507 02:07:01.066392 3422 leader_election.cc:288] T 00000000000000000000000000000000 P b97a91efb55a4cd9904e0476fddacc09 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240507 02:07:01.066462 3427 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b97a91efb55a4cd9904e0476fddacc09 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:07:01.066620 3422 sys_catalog.cc:564] T 00000000000000000000000000000000 P b97a91efb55a4cd9904e0476fddacc09 [sys.catalog]: configured and running, proceeding with master startup.
I20240507 02:07:01.066689 3427 raft_consensus.cc:686] T 00000000000000000000000000000000 P b97a91efb55a4cd9904e0476fddacc09 [term 1 LEADER]: Becoming Leader. State: Replica: b97a91efb55a4cd9904e0476fddacc09, State: Running, Role: LEADER
I20240507 02:07:01.066833 3427 consensus_queue.cc:237] T 00000000000000000000000000000000 P b97a91efb55a4cd9904e0476fddacc09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b97a91efb55a4cd9904e0476fddacc09" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 45979 } }
I20240507 02:07:01.067157 3429 sys_catalog.cc:455] T 00000000000000000000000000000000 P b97a91efb55a4cd9904e0476fddacc09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b97a91efb55a4cd9904e0476fddacc09. Latest consensus state: current_term: 1 leader_uuid: "b97a91efb55a4cd9904e0476fddacc09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b97a91efb55a4cd9904e0476fddacc09" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 45979 } } }
I20240507 02:07:01.067224 3429 sys_catalog.cc:458] T 00000000000000000000000000000000 P b97a91efb55a4cd9904e0476fddacc09 [sys.catalog]: This master's current role is: LEADER
I20240507 02:07:01.067338 3428 sys_catalog.cc:455] T 00000000000000000000000000000000 P b97a91efb55a4cd9904e0476fddacc09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b97a91efb55a4cd9904e0476fddacc09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b97a91efb55a4cd9904e0476fddacc09" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 45979 } } }
I20240507 02:07:01.067379 3428 sys_catalog.cc:458] T 00000000000000000000000000000000 P b97a91efb55a4cd9904e0476fddacc09 [sys.catalog]: This master's current role is: LEADER
I20240507 02:07:01.067744 3436 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 02:07:01.068146 3436 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240507 02:07:01.069422 3436 catalog_manager.cc:1347] Generated new cluster ID: b69a9c7283d64fd3bd0dc2790bb3363d
I20240507 02:07:01.069471 3436 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
May 07 02:07:01 dist-test-slave-v0bg krb5kdc[3341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:07:01.073727 3436 catalog_manager.cc:1370] Generated new certificate authority record
I20240507 02:07:01.074069 3426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:01.064342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47834 (local address 127.31.131.126:45979)
0507 02:07:01.064490 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:01.064499 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:01.064516 (+ 17us) server_negotiation.cc:407] Connection header received
0507 02:07:01.065098 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:01.065103 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:01.065139 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:01.065258 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:01.066134 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:01.067600 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:01.068477 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:01.068656 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:01.070809 (+ 2153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:01.070830 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:01.070831 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:07:01.070848 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:01.072727 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:01.073058 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:01.073060 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:01.073060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:01.073118 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:01.073261 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:01.073262 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:01.073263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:01.073476 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:01.073614 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:01.073748 (+ 134us) server_negotiation.cc:299] Negotiation successful
0507 02:07:01.073875 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":74,"threads_started":1}
I20240507 02:07:01.074522 3436 catalog_manager.cc:1504] Loading token signing keys...
I20240507 02:07:01.080555 3436 catalog_manager.cc:5857] T 00000000000000000000000000000000 P b97a91efb55a4cd9904e0476fddacc09: Generated new TSK 0
I20240507 02:07:01.080701 3436 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.131.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab.
Entry for principal kudu/127.31.131.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab.
I20240507 02:07:01.102932 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.65:0
--local_ip_for_outbound_sockets=127.31.131.65
--webserver_interface=127.31.131.65
--webserver_port=0
--tserver_master_addrs=127.31.131.126:45979
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab
--principal=kudu/127.31.131.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:43217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:07:01.177798 3449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:07:01.177963 3449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:07:01.177984 3449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:07:01.179279 3449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:07:01.179320 3449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:07:01.179344 3449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.65
I20240507 02:07:01.180864 3449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:43217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab
--principal=kudu/127.31.131.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.131.65
--webserver_port=0
--tserver_master_addrs=127.31.131.126:45979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.65
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:07:01.181047 3449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:07:01.181227 3449 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:07:01.184916 3454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:07:01.184937 3455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:07:01.184911 3457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:07:01.185390 3449 server_base.cc:1034] running on GCE node
I20240507 02:07:01.185631 3449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:07:01.185846 3449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:07:01.186968 3449 hybrid_clock.cc:648] HybridClock initialized: now 1715047621186953 us; error 32 us; skew 500 ppm
May 07 02:07:01 dist-test-slave-v0bg krb5kdc[3341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047621, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:07:01.189395 3449 init.cc:377] Logged in from keytab as kudu/127.31.131.65@KRBTEST.COM (short username kudu)
I20240507 02:07:01.189854 3449 webserver.cc:460] Webserver started at http://127.31.131.65:37975/ using document root <none> and password file <none>
I20240507 02:07:01.190060 3449 fs_manager.cc:359] Metadata directory not provided
I20240507 02:07:01.190116 3449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:07:01.190240 3449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:07:01.191498 3449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/minicluster-data/ts-0/data/instance:
uuid: "8b16877875cb4ac3a3907e5719384569"
format_stamp: "Formatted at 2024-05-07 02:07:01 on dist-test-slave-v0bg"
server_key: "df5129f04975c8f4c52ac3f7df843e67"
server_key_iv: "00d0fdcbda5b12b604f54487fb9e4c75"
server_key_version: "encryptionkey@0"
I20240507 02:07:01.191850 3449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/minicluster-data/ts-0/wal/instance:
uuid: "8b16877875cb4ac3a3907e5719384569"
format_stamp: "Formatted at 2024-05-07 02:07:01 on dist-test-slave-v0bg"
server_key: "df5129f04975c8f4c52ac3f7df843e67"
server_key_iv: "00d0fdcbda5b12b604f54487fb9e4c75"
server_key_version: "encryptionkey@0"
I20240507 02:07:01.193239 3449 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240507 02:07:01.194065 3464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:01.194257 3449 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240507 02:07:01.194321 3449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/minicluster-data/ts-0/wal
uuid: "8b16877875cb4ac3a3907e5719384569"
format_stamp: "Formatted at 2024-05-07 02:07:01 on dist-test-slave-v0bg"
server_key: "df5129f04975c8f4c52ac3f7df843e67"
server_key_iv: "00d0fdcbda5b12b604f54487fb9e4c75"
server_key_version: "encryptionkey@0"
I20240507 02:07:01.194383 3449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:07:01.231541 3449 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:07:01.233299 3449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:07:01.233398 3449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:07:01.233628 3449 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:07:01.233942 3449 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:07:01.233974 3449 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:01.233996 3449 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:07:01.234019 3449 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:01.238960 3449 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.65:46003
I20240507 02:07:01.239023 3578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.65:46003 every 8 connection(s)
I20240507 02:07:01.239279 3449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
May 07 02:07:01 dist-test-slave-v0bg krb5kdc[3341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047621, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.65@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:07:01.246378 3426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:01.239895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:44409 (local address 127.31.131.126:45979)
0507 02:07:01.239944 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:01.239946 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:01.240117 (+ 171us) server_negotiation.cc:407] Connection header received
0507 02:07:01.240556 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:01.240556 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:01.240569 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:01.240621 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:01.241336 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:01.241819 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:01.242321 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:01.242432 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:01.244339 (+ 1907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:01.244343 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:01.244343 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:07:01.244349 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:01.245546 (+ 1197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:01.245814 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:01.245815 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:01.245815 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:01.245830 (+ 15us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:01.245950 (+ 120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:01.245950 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:01.245950 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:01.246022 (+ 72us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:01.246061 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:01.246249 (+ 188us) server_negotiation.cc:299] Negotiation successful
0507 02:07:01.246311 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20240507 02:07:01.246470 3581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:01.239862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:45979 (local address 127.31.131.65:44409)
0507 02:07:01.240018 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:01.240043 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:01.240477 (+ 434us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:01.240676 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:01.240679 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:01.240954 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:07:01.241233 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:01.241237 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:01.241890 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:01.241890 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:01.242262 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:01.242263 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:01.242618 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:01.243038 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:07:01.243050 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:07:01.244259 (+ 1209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:07:01.245603 (+ 1344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:01.245606 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:01.245608 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:01.245765 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:01.245869 (+ 104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:01.245870 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:01.245870 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:01.245913 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:01.246059 (+ 146us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:07:01.246061 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:07:01.246152 (+ 91us) client_negotiation.cc:769] Sending connection context
0507 02:07:01.246237 (+ 85us) client_negotiation.cc:240] Negotiation successful
0507 02:07:01.246327 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":83,"threads_started":1}
I20240507 02:07:01.246886 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 3449
I20240507 02:07:01.246953 3579 heartbeater.cc:347] Connected to a master server at 127.31.131.126:45979
I20240507 02:07:01.246986 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/minicluster-data/ts-0/wal/instance
I20240507 02:07:01.247043 3579 heartbeater.cc:464] Registering TS with master...
I20240507 02:07:01.247150 32269 external_mini_cluster.cc:1428] Setting key f57b03da635fe2deef00e9ddf5ae144d
I20240507 02:07:01.247232 3579 heartbeater.cc:511] Master 127.31.131.126:45979 requested a full tablet report, sending...
I20240507 02:07:01.247741 3385 ts_manager.cc:194] Registered new tserver with Master: 8b16877875cb4ac3a3907e5719384569 (127.31.131.65:46003)
I20240507 02:07:01.248399 3385 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.131.65@KRBTEST.COM'} at 127.31.131.65:44409
WARNING: no policy specified for kudu/127.31.131.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab.
Entry for principal kudu/127.31.131.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab.
I20240507 02:07:01.267585 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.66:0
--local_ip_for_outbound_sockets=127.31.131.66
--webserver_interface=127.31.131.66
--webserver_port=0
--tserver_master_addrs=127.31.131.126:45979
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab
--principal=kudu/127.31.131.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:43217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:07:01.342628 3584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:07:01.342797 3584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:07:01.342818 3584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:07:01.344020 3584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:07:01.344056 3584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:07:01.344079 3584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.66
I20240507 02:07:01.345561 3584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:43217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab
--principal=kudu/127.31.131.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.131.66
--webserver_port=0
--tserver_master_addrs=127.31.131.126:45979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.66
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:07:01.345727 3584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:07:01.345872 3584 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240507 02:07:01.349530 3584 server_base.cc:1034] running on GCE node
W20240507 02:07:01.349617 3592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:07:01.349671 3590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:07:01.349653 3589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:07:01.349869 3584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:07:01.350037 3584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:07:01.351153 3584 hybrid_clock.cc:648] HybridClock initialized: now 1715047621351145 us; error 28 us; skew 500 ppm
May 07 02:07:01 dist-test-slave-v0bg krb5kdc[3341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047621, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:07:01.353207 3584 init.cc:377] Logged in from keytab as kudu/127.31.131.66@KRBTEST.COM (short username kudu)
I20240507 02:07:01.353587 3584 webserver.cc:460] Webserver started at http://127.31.131.66:39439/ using document root <none> and password file <none>
I20240507 02:07:01.353739 3584 fs_manager.cc:359] Metadata directory not provided
I20240507 02:07:01.353785 3584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:07:01.353880 3584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:07:01.354832 3584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/minicluster-data/ts-1/data/instance:
uuid: "946c2578f20542bd895044035031c6d0"
format_stamp: "Formatted at 2024-05-07 02:07:01 on dist-test-slave-v0bg"
server_key: "8a55c938d9dd98e93089ba802b5f3480"
server_key_iv: "530d7da2047fa0c9193d8e5cb9931b00"
server_key_version: "encryptionkey@0"
I20240507 02:07:01.355099 3584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/minicluster-data/ts-1/wal/instance:
uuid: "946c2578f20542bd895044035031c6d0"
format_stamp: "Formatted at 2024-05-07 02:07:01 on dist-test-slave-v0bg"
server_key: "8a55c938d9dd98e93089ba802b5f3480"
server_key_iv: "530d7da2047fa0c9193d8e5cb9931b00"
server_key_version: "encryptionkey@0"
I20240507 02:07:01.356137 3584 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240507 02:07:01.356737 3599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:01.356879 3584 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240507 02:07:01.356947 3584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/minicluster-data/ts-1/wal
uuid: "946c2578f20542bd895044035031c6d0"
format_stamp: "Formatted at 2024-05-07 02:07:01 on dist-test-slave-v0bg"
server_key: "8a55c938d9dd98e93089ba802b5f3480"
server_key_iv: "530d7da2047fa0c9193d8e5cb9931b00"
server_key_version: "encryptionkey@0"
I20240507 02:07:01.357002 3584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:07:01.363952 3584 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:07:01.365414 3584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:07:01.365514 3584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:07:01.365670 3584 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:07:01.365938 3584 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:07:01.365968 3584 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:01.365995 3584 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:07:01.366016 3584 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:01.371039 3584 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.66:44119
I20240507 02:07:01.371064 3712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.66:44119 every 8 connection(s)
I20240507 02:07:01.371336 3584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
May 07 02:07:01 dist-test-slave-v0bg krb5kdc[3341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047621, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.66@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:07:01.378199 3426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:01.371917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.66:54337 (local address 127.31.131.126:45979)
0507 02:07:01.371966 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:01.371968 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:01.372106 (+ 138us) server_negotiation.cc:407] Connection header received
0507 02:07:01.372562 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:01.372563 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:01.372575 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:01.372628 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:01.373310 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:01.373807 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:01.374343 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:01.374502 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:01.376289 (+ 1787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:01.376293 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:01.376293 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:07:01.376299 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:01.377335 (+ 1036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:01.377630 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:01.377630 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:01.377631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:01.377646 (+ 15us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:01.377771 (+ 125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:01.377771 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:01.377771 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:01.377848 (+ 77us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:01.377888 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:01.378071 (+ 183us) server_negotiation.cc:299] Negotiation successful
0507 02:07:01.378130 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20240507 02:07:01.378299 3715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:01.371880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:45979 (local address 127.31.131.66:54337)
0507 02:07:01.372018 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:01.372039 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:01.372469 (+ 430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:01.372687 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:01.372690 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:01.372955 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:07:01.373217 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:01.373220 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:01.373879 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:01.373880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:01.374259 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:01.374261 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:01.374578 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:01.374994 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:07:01.375005 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:07:01.376203 (+ 1198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:07:01.377404 (+ 1201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:01.377408 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:01.377410 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:01.377579 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:01.377688 (+ 109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:01.377688 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:01.377689 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:01.377734 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:01.377887 (+ 153us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:07:01.377889 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:07:01.377980 (+ 91us) client_negotiation.cc:769] Sending connection context
0507 02:07:01.378055 (+ 75us) client_negotiation.cc:240] Negotiation successful
0507 02:07:01.378146 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":75,"threads_started":1}
I20240507 02:07:01.378759 3713 heartbeater.cc:347] Connected to a master server at 127.31.131.126:45979
I20240507 02:07:01.378839 3713 heartbeater.cc:464] Registering TS with master...
I20240507 02:07:01.378995 3713 heartbeater.cc:511] Master 127.31.131.126:45979 requested a full tablet report, sending...
I20240507 02:07:01.379315 3385 ts_manager.cc:194] Registered new tserver with Master: 946c2578f20542bd895044035031c6d0 (127.31.131.66:44119)
I20240507 02:07:01.379653 3385 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.131.66@KRBTEST.COM'} at 127.31.131.66:54337
I20240507 02:07:01.381001 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 3584
I20240507 02:07:01.381078 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/minicluster-data/ts-1/wal/instance
I20240507 02:07:01.381209 32269 external_mini_cluster.cc:1428] Setting key a07fe312f3f7b2c31aa390aa01751eaa
WARNING: no policy specified for kudu/127.31.131.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab.
Entry for principal kudu/127.31.131.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab.
I20240507 02:07:01.401947 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.67:0
--local_ip_for_outbound_sockets=127.31.131.67
--webserver_interface=127.31.131.67
--webserver_port=0
--tserver_master_addrs=127.31.131.126:45979
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab
--principal=kudu/127.31.131.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:43217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:07:01.475333 3718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:07:01.475489 3718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:07:01.475518 3718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:07:01.476755 3718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:07:01.476794 3718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:07:01.476826 3718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.67
I20240507 02:07:01.478322 3718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:43217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab
--principal=kudu/127.31.131.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.131.67
--webserver_port=0
--tserver_master_addrs=127.31.131.126:45979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3718
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.67
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:07:01.478521 3718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:07:01.478709 3718 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:07:01.482460 3726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:07:01.482460 3724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:07:01.482621 3723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:07:01.482583 3718 server_base.cc:1034] running on GCE node
I20240507 02:07:01.482795 3718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:07:01.482962 3718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:07:01.484076 3718 hybrid_clock.cc:648] HybridClock initialized: now 1715047621484068 us; error 25 us; skew 500 ppm
May 07 02:07:01 dist-test-slave-v0bg krb5kdc[3341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047621, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:07:01.486198 3718 init.cc:377] Logged in from keytab as kudu/127.31.131.67@KRBTEST.COM (short username kudu)
I20240507 02:07:01.486575 3718 webserver.cc:460] Webserver started at http://127.31.131.67:45199/ using document root <none> and password file <none>
I20240507 02:07:01.486743 3718 fs_manager.cc:359] Metadata directory not provided
I20240507 02:07:01.486780 3718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:07:01.486877 3718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:07:01.487784 3718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/minicluster-data/ts-2/data/instance:
uuid: "4a1d00cd6d7a4052a8c6a0d9562566b6"
format_stamp: "Formatted at 2024-05-07 02:07:01 on dist-test-slave-v0bg"
server_key: "c3fc72569b8a22cd8ef184287e7ca04b"
server_key_iv: "1a5db28044794a064ef2b5dd7138cd3a"
server_key_version: "encryptionkey@0"
I20240507 02:07:01.488057 3718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/minicluster-data/ts-2/wal/instance:
uuid: "4a1d00cd6d7a4052a8c6a0d9562566b6"
format_stamp: "Formatted at 2024-05-07 02:07:01 on dist-test-slave-v0bg"
server_key: "c3fc72569b8a22cd8ef184287e7ca04b"
server_key_iv: "1a5db28044794a064ef2b5dd7138cd3a"
server_key_version: "encryptionkey@0"
I20240507 02:07:01.489075 3718 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240507 02:07:01.489609 3733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:01.489764 3718 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240507 02:07:01.489841 3718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/minicluster-data/ts-2/wal
uuid: "4a1d00cd6d7a4052a8c6a0d9562566b6"
format_stamp: "Formatted at 2024-05-07 02:07:01 on dist-test-slave-v0bg"
server_key: "c3fc72569b8a22cd8ef184287e7ca04b"
server_key_iv: "1a5db28044794a064ef2b5dd7138cd3a"
server_key_version: "encryptionkey@0"
I20240507 02:07:01.489894 3718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:07:01.498004 3718 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:07:01.499912 3718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:07:01.500031 3718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:07:01.500202 3718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:07:01.500468 3718 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:07:01.500499 3718 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:01.500524 3718 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:07:01.500540 3718 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:01.505442 3718 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.67:41871
I20240507 02:07:01.505474 3847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.67:41871 every 8 connection(s)
I20240507 02:07:01.505736 3718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
May 07 02:07:01 dist-test-slave-v0bg krb5kdc[3341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047621, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.67@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:07:01.512821 3426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:01.506336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.67:43603 (local address 127.31.131.126:45979)
0507 02:07:01.506388 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:01.506390 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:01.506536 (+ 146us) server_negotiation.cc:407] Connection header received
0507 02:07:01.506996 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:01.506997 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:01.507005 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:01.507059 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:01.507728 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:01.508207 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:01.508687 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:01.508797 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:01.510796 (+ 1999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:01.510801 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:01.510802 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:07:01.510808 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:01.511992 (+ 1184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:01.512269 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:01.512270 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:01.512270 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:01.512285 (+ 15us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:01.512398 (+ 113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:01.512398 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:01.512398 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:01.512475 (+ 77us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:01.512514 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:01.512685 (+ 171us) server_negotiation.cc:299] Negotiation successful
0507 02:07:01.512756 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20240507 02:07:01.512920 3850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:01.506303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:45979 (local address 127.31.131.67:43603)
0507 02:07:01.506437 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:01.506465 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:01.506919 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:01.507112 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:01.507115 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:01.507392 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:07:01.507655 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:01.507659 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:01.508269 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:01.508269 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:01.508634 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:01.508635 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:01.508934 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:01.509375 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:07:01.509388 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:07:01.510715 (+ 1327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:07:01.512055 (+ 1340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:01.512058 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:01.512061 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:01.512221 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:01.512325 (+ 104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:01.512325 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:01.512326 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:01.512368 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:01.512512 (+ 144us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:07:01.512514 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:07:01.512605 (+ 91us) client_negotiation.cc:769] Sending connection context
0507 02:07:01.512678 (+ 73us) client_negotiation.cc:240] Negotiation successful
0507 02:07:01.512774 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":58,"threads_started":1}
I20240507 02:07:01.513343 3848 heartbeater.cc:347] Connected to a master server at 127.31.131.126:45979
I20240507 02:07:01.513423 3848 heartbeater.cc:464] Registering TS with master...
I20240507 02:07:01.513603 3848 heartbeater.cc:511] Master 127.31.131.126:45979 requested a full tablet report, sending...
I20240507 02:07:01.513940 3385 ts_manager.cc:194] Registered new tserver with Master: 4a1d00cd6d7a4052a8c6a0d9562566b6 (127.31.131.67:41871)
I20240507 02:07:01.514287 3385 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.131.67@KRBTEST.COM'} at 127.31.131.67:43603
I20240507 02:07:01.515323 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 3718
I20240507 02:07:01.515395 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1715047604443657-32269-0/minicluster-data/ts-2/wal/instance
I20240507 02:07:01.515543 32269 external_mini_cluster.cc:1428] Setting key e9d6587cb1a008e7a4dbae0254568a61
I20240507 02:07:01.516263 32269 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
May 07 02:07:01 dist-test-slave-v0bg krb5kdc[3341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047621, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
May 07 02:07:01 dist-test-slave-v0bg krb5kdc[3341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047621, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.131.65@KRBTEST.COM
I20240507 02:07:01.536360 3855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:01.529064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50542 (local address 127.31.131.65:46003)
0507 02:07:01.529204 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:01.529211 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:01.529220 (+ 9us) server_negotiation.cc:407] Connection header received
0507 02:07:01.529245 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:01.529246 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:01.529259 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:01.529323 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:01.529910 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:01.530625 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:01.531139 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:01.531349 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:01.533589 (+ 2240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:01.533601 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:01.533602 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:07:01.533607 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:01.535315 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:01.535595 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:01.535598 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:01.535598 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:01.535634 (+ 36us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:01.535788 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:01.535788 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:01.535789 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:01.535896 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:01.536034 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:01.536164 (+ 130us) server_negotiation.cc:299] Negotiation successful
0507 02:07:01.536250 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":64,"threads_started":1}
May 07 02:07:01 dist-test-slave-v0bg krb5kdc[3341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047621, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:07:01.543359 3426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:01.538046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47844 (local address 127.31.131.126:45979)
0507 02:07:01.538083 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:01.538085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:01.538130 (+ 45us) server_negotiation.cc:407] Connection header received
0507 02:07:01.538176 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:01.538177 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:01.538185 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:01.538232 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:01.538746 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:01.539198 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:01.539647 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:01.539747 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:01.541399 (+ 1652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:01.541403 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:01.541403 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:07:01.541407 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:01.542466 (+ 1059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:01.542759 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:01.542761 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:01.542761 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:01.542785 (+ 24us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:01.542904 (+ 119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:01.542904 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:01.542905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:01.543009 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:01.543097 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:01.543190 (+ 93us) server_negotiation.cc:299] Negotiation successful
0507 02:07:01.543269 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20240507 02:07:01.543562 3385 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47844
I20240507 02:07:01.543929 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 3449
I20240507 02:07:01.548336 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 3584
I20240507 02:07:01.552102 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 3718
I20240507 02:07:01.555675 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 3354
2024-05-07T02:07:01Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1616 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 07 02:07:01 dist-test-slave-v0bg krb5kdc[3860](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 07 02:07:01 dist-test-slave-v0bg krb5kdc[3860](info): set up 2 sockets
May 07 02:07:01 dist-test-slave-v0bg krb5kdc[3860](info): commencing operation
krb5kdc: starting...
W20240507 02:07:02.439769 32269 mini_kdc.cc:121] Time spent starting KDC: real 0.866s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/krb5kdc/test-user.keytab.
May 07 02:07:02 dist-test-slave-v0bg krb5kdc[3860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-07T02:07:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-07T02:07:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.131.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab.
Entry for principal kudu/127.31.131.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab.
I20240507 02:07:02.539422 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:44301
--webserver_interface=127.31.131.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab
--principal=kudu/127.31.131.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:38895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.131.126:44301
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:07:02.613749 3873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:07:02.613909 3873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:07:02.613930 3873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:07:02.615224 3873 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240507 02:07:02.615270 3873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:07:02.615283 3873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:07:02.615304 3873 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240507 02:07:02.615327 3873 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240507 02:07:02.616868 3873 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:38895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.131.126:44301
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab
--principal=kudu/127.31.131.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:44301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.131.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:07:02.617064 3873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:07:02.617229 3873 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:07:02.620682 3879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:07:02.620712 3873 server_base.cc:1034] running on GCE node
W20240507 02:07:02.620684 3880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:07:02.620669 3882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:07:02.621084 3873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:07:02.621307 3873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:07:02.622424 3873 hybrid_clock.cc:648] HybridClock initialized: now 1715047622622413 us; error 33 us; skew 500 ppm
May 07 02:07:02 dist-test-slave-v0bg krb5kdc[3860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047622, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:07:02.624505 3873 init.cc:377] Logged in from keytab as kudu/127.31.131.126@KRBTEST.COM (short username kudu)
I20240507 02:07:02.624797 3873 fs_manager.cc:359] Metadata directory not provided
I20240507 02:07:02.624842 3873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:07:02.624938 3873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:07:02.625900 3873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/minicluster-data/master-0/data/instance:
uuid: "d1760ccb8b9143af83a5684793dcd5ca"
format_stamp: "Formatted at 2024-05-07 02:07:02 on dist-test-slave-v0bg"
server_key: "519c95c4ad0bdd6b33ccd9204ae3e8e2"
server_key_iv: "1bf6b92d00372a19db603e0ed5396a2f"
server_key_version: "encryptionkey@0"
I20240507 02:07:02.626178 3873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/minicluster-data/master-0/wal/instance:
uuid: "d1760ccb8b9143af83a5684793dcd5ca"
format_stamp: "Formatted at 2024-05-07 02:07:02 on dist-test-slave-v0bg"
server_key: "519c95c4ad0bdd6b33ccd9204ae3e8e2"
server_key_iv: "1bf6b92d00372a19db603e0ed5396a2f"
server_key_version: "encryptionkey@0"
I20240507 02:07:02.627250 3873 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20240507 02:07:02.627825 3888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:02.627972 3873 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:02.628041 3873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/minicluster-data/master-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/minicluster-data/master-0/wal
uuid: "d1760ccb8b9143af83a5684793dcd5ca"
format_stamp: "Formatted at 2024-05-07 02:07:02 on dist-test-slave-v0bg"
server_key: "519c95c4ad0bdd6b33ccd9204ae3e8e2"
server_key_iv: "1bf6b92d00372a19db603e0ed5396a2f"
server_key_version: "encryptionkey@0"
I20240507 02:07:02.628096 3873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:07:02.637590 3873 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:07:02.639036 3873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:07:02.639139 3873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:07:02.642828 3873 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.126:44301
I20240507 02:07:02.642853 3940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.126:44301 every 8 connection(s)
I20240507 02:07:02.643082 3873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
I20240507 02:07:02.643564 3941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:07:02.645900 3941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1760ccb8b9143af83a5684793dcd5ca: Bootstrap starting.
I20240507 02:07:02.646392 3941 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1760ccb8b9143af83a5684793dcd5ca: Neither blocks nor log segments found. Creating new log.
I20240507 02:07:02.646651 3941 log.cc:824] T 00000000000000000000000000000000 P d1760ccb8b9143af83a5684793dcd5ca: Log is configured to *not* fsync() on all Append() calls
I20240507 02:07:02.647213 3941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1760ccb8b9143af83a5684793dcd5ca: No bootstrap required, opened a new log
I20240507 02:07:02.648559 3941 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1760ccb8b9143af83a5684793dcd5ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1760ccb8b9143af83a5684793dcd5ca" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 44301 } }
I20240507 02:07:02.648679 3941 raft_consensus.cc:374] T 00000000000000000000000000000000 P d1760ccb8b9143af83a5684793dcd5ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:07:02.648694 3941 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1760ccb8b9143af83a5684793dcd5ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1760ccb8b9143af83a5684793dcd5ca, State: Initialized, Role: FOLLOWER
I20240507 02:07:02.648836 3941 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1760ccb8b9143af83a5684793dcd5ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1760ccb8b9143af83a5684793dcd5ca" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 44301 } }
I20240507 02:07:02.648897 3941 raft_consensus.cc:388] T 00000000000000000000000000000000 P d1760ccb8b9143af83a5684793dcd5ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 02:07:02.648917 3941 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1760ccb8b9143af83a5684793dcd5ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 02:07:02.648957 3941 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1760ccb8b9143af83a5684793dcd5ca [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:07:02.649438 3941 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1760ccb8b9143af83a5684793dcd5ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1760ccb8b9143af83a5684793dcd5ca" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 44301 } }
I20240507 02:07:02.649523 3941 leader_election.cc:302] T 00000000000000000000000000000000 P d1760ccb8b9143af83a5684793dcd5ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1760ccb8b9143af83a5684793dcd5ca; no voters:
I20240507 02:07:02.649659 3941 leader_election.cc:288] T 00000000000000000000000000000000 P d1760ccb8b9143af83a5684793dcd5ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20240507 02:07:02.649710 3944 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1760ccb8b9143af83a5684793dcd5ca [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:07:02.649861 3944 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1760ccb8b9143af83a5684793dcd5ca [term 1 LEADER]: Becoming Leader. State: Replica: d1760ccb8b9143af83a5684793dcd5ca, State: Running, Role: LEADER
I20240507 02:07:02.649905 3941 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1760ccb8b9143af83a5684793dcd5ca [sys.catalog]: configured and running, proceeding with master startup.
I20240507 02:07:02.649941 3944 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1760ccb8b9143af83a5684793dcd5ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1760ccb8b9143af83a5684793dcd5ca" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 44301 } }
I20240507 02:07:02.650346 3946 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1760ccb8b9143af83a5684793dcd5ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1760ccb8b9143af83a5684793dcd5ca. Latest consensus state: current_term: 1 leader_uuid: "d1760ccb8b9143af83a5684793dcd5ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1760ccb8b9143af83a5684793dcd5ca" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 44301 } } }
I20240507 02:07:02.650496 3946 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1760ccb8b9143af83a5684793dcd5ca [sys.catalog]: This master's current role is: LEADER
I20240507 02:07:02.650498 3945 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1760ccb8b9143af83a5684793dcd5ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d1760ccb8b9143af83a5684793dcd5ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1760ccb8b9143af83a5684793dcd5ca" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 44301 } } }
I20240507 02:07:02.650700 3945 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1760ccb8b9143af83a5684793dcd5ca [sys.catalog]: This master's current role is: LEADER
W20240507 02:07:02.651517 3960 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d1760ccb8b9143af83a5684793dcd5ca: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240507 02:07:02.651571 3960 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240507 02:07:02.651652 3953 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 02:07:02.651834 3953 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240507 02:07:02.652877 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 3873
I20240507 02:07:02.652981 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/minicluster-data/master-0/wal/instance
I20240507 02:07:02.653059 3953 catalog_manager.cc:1347] Generated new cluster ID: 6102f83c81674e2e9d61facf3b22f87f
I20240507 02:07:02.653095 3953 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240507 02:07:02.653146 32269 external_mini_cluster.cc:1428] Setting key 7bb6bfee8721f74119e6f30a60c9c2c8
May 07 02:07:02 dist-test-slave-v0bg krb5kdc[3860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:07:02.660152 3953 catalog_manager.cc:1370] Generated new certificate authority record
I20240507 02:07:02.660687 3953 catalog_manager.cc:1504] Loading token signing keys...
I20240507 02:07:02.662987 3964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:02.653610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42700 (local address 127.31.131.126:44301)
0507 02:07:02.653763 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:02.653776 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:02.653965 (+ 189us) server_negotiation.cc:407] Connection header received
0507 02:07:02.654071 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:02.654077 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:02.654123 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:02.654249 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:02.655146 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:02.656282 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:02.657045 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:02.657578 (+ 533us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:02.659638 (+ 2060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:02.659644 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:02.659645 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:07:02.659663 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:02.661676 (+ 2013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:02.662018 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:02.662021 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:02.662022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:02.662079 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:02.662214 (+ 135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:02.662215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:02.662215 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:02.662428 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:02.662587 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:02.662661 (+ 74us) server_negotiation.cc:299] Negotiation successful
0507 02:07:02.662801 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":71,"threads_started":1}
I20240507 02:07:02.664113 3953 catalog_manager.cc:5857] T 00000000000000000000000000000000 P d1760ccb8b9143af83a5684793dcd5ca: Generated new TSK 0
I20240507 02:07:02.664245 3953 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.131.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab.
Entry for principal kudu/127.31.131.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab.
I20240507 02:07:02.691049 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.65:0
--local_ip_for_outbound_sockets=127.31.131.65
--webserver_interface=127.31.131.65
--webserver_port=0
--tserver_master_addrs=127.31.131.126:44301
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab
--principal=kudu/127.31.131.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:38895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:07:02.767112 3967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:07:02.767282 3967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:07:02.767302 3967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:07:02.768566 3967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:07:02.768610 3967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:07:02.768632 3967 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.65
I20240507 02:07:02.770184 3967 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:38895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab
--principal=kudu/127.31.131.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.131.65
--webserver_port=0
--tserver_master_addrs=127.31.131.126:44301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.65
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:07:02.770363 3967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:07:02.770565 3967 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:07:02.774715 3972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:07:02.774869 3967 server_base.cc:1034] running on GCE node
W20240507 02:07:02.774716 3973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:07:02.774776 3975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:07:02.775074 3967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:07:02.775272 3967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:07:02.776403 3967 hybrid_clock.cc:648] HybridClock initialized: now 1715047622776391 us; error 29 us; skew 500 ppm
May 07 02:07:02 dist-test-slave-v0bg krb5kdc[3860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047622, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:07:02.778632 3967 init.cc:377] Logged in from keytab as kudu/127.31.131.65@KRBTEST.COM (short username kudu)
I20240507 02:07:02.778890 3967 fs_manager.cc:359] Metadata directory not provided
I20240507 02:07:02.778929 3967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:07:02.779014 3967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:07:02.779925 3967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/minicluster-data/ts-0/data/instance:
uuid: "a83ba02744de47499ab6259aa3f8c3a1"
format_stamp: "Formatted at 2024-05-07 02:07:02 on dist-test-slave-v0bg"
server_key: "737128944af02ade109b4bfe1afbc39f"
server_key_iv: "3d99c7f4ee078e02ea237404083b1c02"
server_key_version: "encryptionkey@0"
I20240507 02:07:02.780194 3967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/minicluster-data/ts-0/wal/instance:
uuid: "a83ba02744de47499ab6259aa3f8c3a1"
format_stamp: "Formatted at 2024-05-07 02:07:02 on dist-test-slave-v0bg"
server_key: "737128944af02ade109b4bfe1afbc39f"
server_key_iv: "3d99c7f4ee078e02ea237404083b1c02"
server_key_version: "encryptionkey@0"
I20240507 02:07:02.781239 3967 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240507 02:07:02.781862 3981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:02.782048 3967 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240507 02:07:02.782119 3967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/minicluster-data/ts-0/wal
uuid: "a83ba02744de47499ab6259aa3f8c3a1"
format_stamp: "Formatted at 2024-05-07 02:07:02 on dist-test-slave-v0bg"
server_key: "737128944af02ade109b4bfe1afbc39f"
server_key_iv: "3d99c7f4ee078e02ea237404083b1c02"
server_key_version: "encryptionkey@0"
I20240507 02:07:02.782179 3967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:07:02.814162 3967 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:07:02.815775 3967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:07:02.815883 3967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:07:02.816097 3967 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:07:02.816391 3967 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:07:02.816422 3967 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:02.816452 3967 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:07:02.816473 3967 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:02.821473 3967 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.65:44405
I20240507 02:07:02.821521 4095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.65:44405 every 8 connection(s)
I20240507 02:07:02.821725 3967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
I20240507 02:07:02.824901 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 3967
I20240507 02:07:02.824995 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/minicluster-data/ts-0/wal/instance
I20240507 02:07:02.825158 32269 external_mini_cluster.cc:1428] Setting key 595b02be60da00f43ab161d430d1e9b5
May 07 02:07:02 dist-test-slave-v0bg krb5kdc[3860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047622, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.65@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:07:02.829391 3964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:02.822285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:42937 (local address 127.31.131.126:44301)
0507 02:07:02.822333 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:02.822334 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:02.822483 (+ 149us) server_negotiation.cc:407] Connection header received
0507 02:07:02.823001 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:02.823002 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:02.823014 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:02.823069 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:02.823762 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:02.824250 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:02.824773 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:02.824953 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:02.826967 (+ 2014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:02.826971 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:02.826971 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:07:02.826977 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:02.828311 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:02.828643 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:02.828644 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:02.828645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:02.828669 (+ 24us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:02.828842 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:02.828842 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:02.828843 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:02.828948 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:02.829012 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:02.829210 (+ 198us) server_negotiation.cc:299] Negotiation successful
0507 02:07:02.829302 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20240507 02:07:02.829406 4098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:02.822262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:44301 (local address 127.31.131.65:42937)
0507 02:07:02.822386 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:02.822412 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:02.822900 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:02.823127 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:02.823130 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:02.823408 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:07:02.823670 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:02.823674 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:02.824323 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:02.824324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:02.824680 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:02.824681 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:02.825088 (+ 407us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:02.825508 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:07:02.825524 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:07:02.826884 (+ 1360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:07:02.828388 (+ 1504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:02.828393 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:02.828395 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:02.828577 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:02.828731 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:02.828732 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:02.828732 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:02.828789 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:02.829003 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:07:02.829006 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:07:02.829103 (+ 97us) client_negotiation.cc:769] Sending connection context
0507 02:07:02.829178 (+ 75us) client_negotiation.cc:240] Negotiation successful
0507 02:07:02.829278 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":65,"threads_started":1}
I20240507 02:07:02.829933 4096 heartbeater.cc:347] Connected to a master server at 127.31.131.126:44301
I20240507 02:07:02.830029 4096 heartbeater.cc:464] Registering TS with master...
I20240507 02:07:02.830204 4096 heartbeater.cc:511] Master 127.31.131.126:44301 requested a full tablet report, sending...
I20240507 02:07:02.830714 3905 ts_manager.cc:194] Registered new tserver with Master: a83ba02744de47499ab6259aa3f8c3a1 (127.31.131.65:44405)
I20240507 02:07:02.831642 3905 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.131.65@KRBTEST.COM'} at 127.31.131.65:42937
WARNING: no policy specified for kudu/127.31.131.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab.
Entry for principal kudu/127.31.131.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab.
I20240507 02:07:02.847147 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.66:0
--local_ip_for_outbound_sockets=127.31.131.66
--webserver_interface=127.31.131.66
--webserver_port=0
--tserver_master_addrs=127.31.131.126:44301
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab
--principal=kudu/127.31.131.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:38895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:07:02.922837 4101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:07:02.922998 4101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:07:02.923018 4101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:07:02.924237 4101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:07:02.924275 4101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:07:02.924299 4101 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.66
I20240507 02:07:02.925856 4101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:38895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab
--principal=kudu/127.31.131.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.131.66
--webserver_port=0
--tserver_master_addrs=127.31.131.126:44301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.66
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:07:02.926025 4101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:07:02.926182 4101 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:07:02.929822 4109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:07:02.929931 4106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:07:02.929867 4101 server_base.cc:1034] running on GCE node
W20240507 02:07:02.930089 4107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:07:02.930250 4101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:07:02.930478 4101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:07:02.931607 4101 hybrid_clock.cc:648] HybridClock initialized: now 1715047622931594 us; error 24 us; skew 500 ppm
May 07 02:07:02 dist-test-slave-v0bg krb5kdc[3860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047622, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:07:02.933701 4101 init.cc:377] Logged in from keytab as kudu/127.31.131.66@KRBTEST.COM (short username kudu)
I20240507 02:07:02.933995 4101 fs_manager.cc:359] Metadata directory not provided
I20240507 02:07:02.934039 4101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:07:02.934140 4101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:07:02.935097 4101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/minicluster-data/ts-1/data/instance:
uuid: "054cfd1fcdb3401785582993601b9e0b"
format_stamp: "Formatted at 2024-05-07 02:07:02 on dist-test-slave-v0bg"
server_key: "d20d7a7088aaad896789480fcb1c7560"
server_key_iv: "42a0d9de9048257934b957270785a827"
server_key_version: "encryptionkey@0"
I20240507 02:07:02.935391 4101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/minicluster-data/ts-1/wal/instance:
uuid: "054cfd1fcdb3401785582993601b9e0b"
format_stamp: "Formatted at 2024-05-07 02:07:02 on dist-test-slave-v0bg"
server_key: "d20d7a7088aaad896789480fcb1c7560"
server_key_iv: "42a0d9de9048257934b957270785a827"
server_key_version: "encryptionkey@0"
I20240507 02:07:02.936435 4101 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240507 02:07:02.937016 4115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:02.937155 4101 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240507 02:07:02.937224 4101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/minicluster-data/ts-1/wal
uuid: "054cfd1fcdb3401785582993601b9e0b"
format_stamp: "Formatted at 2024-05-07 02:07:02 on dist-test-slave-v0bg"
server_key: "d20d7a7088aaad896789480fcb1c7560"
server_key_iv: "42a0d9de9048257934b957270785a827"
server_key_version: "encryptionkey@0"
I20240507 02:07:02.937278 4101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:07:02.957880 4101 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:07:02.959540 4101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:07:02.959647 4101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:07:02.959836 4101 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:07:02.960127 4101 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:07:02.960160 4101 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:02.960189 4101 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:07:02.960215 4101 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:02.965181 4101 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.66:38921
I20240507 02:07:02.965225 4229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.66:38921 every 8 connection(s)
I20240507 02:07:02.965440 4101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
May 07 02:07:02 dist-test-slave-v0bg krb5kdc[3860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047622, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.66@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:07:02.970971 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 4101
I20240507 02:07:02.971079 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/minicluster-data/ts-1/wal/instance
I20240507 02:07:02.971251 32269 external_mini_cluster.cc:1428] Setting key f827505aa28087a34da36225e1365f4a
I20240507 02:07:02.972810 3964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:02.966057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.66:41839 (local address 127.31.131.126:44301)
0507 02:07:02.966115 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:02.966116 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:02.966253 (+ 137us) server_negotiation.cc:407] Connection header received
0507 02:07:02.966751 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:02.966752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:02.966776 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:02.966816 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:02.967472 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:02.967958 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:02.968449 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:02.968548 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:02.970413 (+ 1865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:02.970417 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:02.970418 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:07:02.970424 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:02.971781 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:02.972096 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:02.972097 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:02.972097 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:02.972119 (+ 22us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:02.972272 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:02.972272 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:02.972273 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:02.972357 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:02.972409 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:02.972626 (+ 217us) server_negotiation.cc:299] Negotiation successful
0507 02:07:02.972711 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20240507 02:07:02.972900 4232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:02.966015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:44301 (local address 127.31.131.66:41839)
0507 02:07:02.966155 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:02.966182 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:02.966650 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:02.966863 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:02.966867 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:02.967138 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:07:02.967408 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:02.967411 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:02.968028 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:02.968029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:02.968396 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:02.968397 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:02.968680 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:02.969110 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:07:02.969121 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:07:02.970337 (+ 1216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:07:02.971847 (+ 1510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:02.971852 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:02.971855 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:02.972035 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:02.972168 (+ 133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:02.972169 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:02.972169 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:02.972220 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:02.972406 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:07:02.972410 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:07:02.972515 (+ 105us) client_negotiation.cc:769] Sending connection context
0507 02:07:02.972605 (+ 90us) client_negotiation.cc:240] Negotiation successful
0507 02:07:02.972726 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":79,"threads_started":1}
I20240507 02:07:02.973434 4230 heartbeater.cc:347] Connected to a master server at 127.31.131.126:44301
I20240507 02:07:02.973527 4230 heartbeater.cc:464] Registering TS with master...
I20240507 02:07:02.973680 4230 heartbeater.cc:511] Master 127.31.131.126:44301 requested a full tablet report, sending...
I20240507 02:07:02.974027 3905 ts_manager.cc:194] Registered new tserver with Master: 054cfd1fcdb3401785582993601b9e0b (127.31.131.66:38921)
I20240507 02:07:02.974490 3905 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.131.66@KRBTEST.COM'} at 127.31.131.66:41839
WARNING: no policy specified for kudu/127.31.131.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab.
Entry for principal kudu/127.31.131.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab.
I20240507 02:07:02.992996 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.67:0
--local_ip_for_outbound_sockets=127.31.131.67
--webserver_interface=127.31.131.67
--webserver_port=0
--tserver_master_addrs=127.31.131.126:44301
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab
--principal=kudu/127.31.131.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:38895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:07:03.067788 4235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:07:03.067948 4235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:07:03.067970 4235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:07:03.069165 4235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:07:03.069198 4235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:07:03.069221 4235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.67
I20240507 02:07:03.070722 4235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:38895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab
--principal=kudu/127.31.131.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.131.67
--webserver_port=0
--tserver_master_addrs=127.31.131.126:44301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4235
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.67
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:07:03.070887 4235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:07:03.071017 4235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240507 02:07:03.074633 4235 server_base.cc:1034] running on GCE node
W20240507 02:07:03.074666 4240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:07:03.074728 4241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:07:03.074839 4243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:07:03.075013 4235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:07:03.075203 4235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:07:03.076320 4235 hybrid_clock.cc:648] HybridClock initialized: now 1715047623076312 us; error 31 us; skew 500 ppm
May 07 02:07:03 dist-test-slave-v0bg krb5kdc[3860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047623, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:07:03.078424 4235 init.cc:377] Logged in from keytab as kudu/127.31.131.67@KRBTEST.COM (short username kudu)
I20240507 02:07:03.078712 4235 fs_manager.cc:359] Metadata directory not provided
I20240507 02:07:03.078768 4235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:07:03.078869 4235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:07:03.079756 4235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/minicluster-data/ts-2/data/instance:
uuid: "c0c62a5b42404d9aa35dff74b432f3a0"
format_stamp: "Formatted at 2024-05-07 02:07:03 on dist-test-slave-v0bg"
server_key: "aac2f7a4463f6c982aabbd4a38f93342"
server_key_iv: "577fd3f381e8fce3c81d2f73aa463004"
server_key_version: "encryptionkey@0"
I20240507 02:07:03.080036 4235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/minicluster-data/ts-2/wal/instance:
uuid: "c0c62a5b42404d9aa35dff74b432f3a0"
format_stamp: "Formatted at 2024-05-07 02:07:03 on dist-test-slave-v0bg"
server_key: "aac2f7a4463f6c982aabbd4a38f93342"
server_key_iv: "577fd3f381e8fce3c81d2f73aa463004"
server_key_version: "encryptionkey@0"
I20240507 02:07:03.081037 4235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240507 02:07:03.081595 4249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:03.081737 4235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240507 02:07:03.081825 4235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/minicluster-data/ts-2/wal
uuid: "c0c62a5b42404d9aa35dff74b432f3a0"
format_stamp: "Formatted at 2024-05-07 02:07:03 on dist-test-slave-v0bg"
server_key: "aac2f7a4463f6c982aabbd4a38f93342"
server_key_iv: "577fd3f381e8fce3c81d2f73aa463004"
server_key_version: "encryptionkey@0"
I20240507 02:07:03.081881 4235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:07:03.104676 4235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:07:03.106330 4235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:07:03.106453 4235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:07:03.106702 4235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:07:03.107024 4235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:07:03.107061 4235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:03.107095 4235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:07:03.107118 4235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:03.112341 4235 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.67:43513
I20240507 02:07:03.112412 4363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.67:43513 every 8 connection(s)
I20240507 02:07:03.112623 4235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
I20240507 02:07:03.116827 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 4235
I20240507 02:07:03.116912 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/minicluster-data/ts-2/wal/instance
I20240507 02:07:03.117079 32269 external_mini_cluster.cc:1428] Setting key 80e8dd8e6c1546b20081976012d31968
May 07 02:07:03 dist-test-slave-v0bg krb5kdc[3860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047623, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.67@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:07:03.120440 3964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:03.113314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.67:54533 (local address 127.31.131.126:44301)
0507 02:07:03.113370 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:03.113372 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:03.113580 (+ 208us) server_negotiation.cc:407] Connection header received
0507 02:07:03.114036 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:03.114037 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:03.114046 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:03.114102 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:03.114894 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:03.115389 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:03.115935 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:03.116058 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:03.118235 (+ 2177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:03.118238 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:03.118238 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:07:03.118244 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:03.119520 (+ 1276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:03.119844 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:03.119845 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:03.119846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:03.119862 (+ 16us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:03.119997 (+ 135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:03.119997 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:03.119997 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:03.120074 (+ 77us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:03.120115 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:03.120313 (+ 198us) server_negotiation.cc:299] Negotiation successful
0507 02:07:03.120375 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20240507 02:07:03.120580 4366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:03.113295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:44301 (local address 127.31.131.67:54533)
0507 02:07:03.113463 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:03.113494 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:03.113952 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:03.114151 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:03.114154 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:03.114444 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:07:03.114781 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:03.114786 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:03.115463 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:03.115464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:03.115861 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:03.115862 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:03.116161 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:03.116616 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:07:03.116628 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:07:03.118167 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:07:03.119585 (+ 1418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:03.119590 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:03.119593 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:03.119787 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:03.119907 (+ 120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:03.119907 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:03.119908 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:03.119957 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:03.120114 (+ 157us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:07:03.120117 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:07:03.120219 (+ 102us) client_negotiation.cc:769] Sending connection context
0507 02:07:03.120302 (+ 83us) client_negotiation.cc:240] Negotiation successful
0507 02:07:03.120405 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":71,"threads_started":1}
I20240507 02:07:03.121050 4364 heartbeater.cc:347] Connected to a master server at 127.31.131.126:44301
I20240507 02:07:03.121148 4364 heartbeater.cc:464] Registering TS with master...
I20240507 02:07:03.121327 4364 heartbeater.cc:511] Master 127.31.131.126:44301 requested a full tablet report, sending...
I20240507 02:07:03.121696 3905 ts_manager.cc:194] Registered new tserver with Master: c0c62a5b42404d9aa35dff74b432f3a0 (127.31.131.67:43513)
I20240507 02:07:03.122053 3905 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.131.67@KRBTEST.COM'} at 127.31.131.67:54533
I20240507 02:07:03.128728 32269 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240507 02:07:03.134852 3964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:03.129944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42708 (local address 127.31.131.126:44301)
0507 02:07:03.129991 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:03.129993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:03.130094 (+ 101us) server_negotiation.cc:407] Connection header received
0507 02:07:03.130148 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:03.130149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:03.130159 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:03.130219 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:03.130947 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:03.131495 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:03.132016 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:03.132169 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:03.132999 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:03.133003 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:03.133003 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:07:03.133009 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:03.133986 (+ 977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:03.134239 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:03.134240 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:03.134241 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:03.134260 (+ 19us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:03.134383 (+ 123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:03.134383 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:03.134384 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:03.134490 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:03.134594 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:03.134686 (+ 92us) server_negotiation.cc:299] Negotiation successful
0507 02:07:03.134761 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20240507 02:07:03.136183 3905 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42708:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 02:07:03.137571 3905 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 02:07:03.142053 4372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:03.139384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.66:38921 (local address 127.0.0.1:60732)
0507 02:07:03.139522 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:03.139548 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:03.139609 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:03.139922 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:03.139925 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:03.139928 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:07:03.140195 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:03.140197 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:03.141212 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:03.141213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:03.141790 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:03.141791 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:03.141852 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:03.141892 (+ 40us) client_negotiation.cc:769] Sending connection context
0507 02:07:03.141941 (+ 49us) client_negotiation.cc:240] Negotiation successful
0507 02:07:03.141981 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":83,"threads_started":1}
I20240507 02:07:03.142541 4373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:03.139483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60732 (local address 127.31.131.66:38921)
0507 02:07:03.139646 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:03.139653 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:03.139663 (+ 10us) server_negotiation.cc:407] Connection header received
0507 02:07:03.139700 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:03.139702 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:03.139718 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:03.139804 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:07:03.140254 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:03.141148 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:03.141881 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:03.142359 (+ 478us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:03.142410 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:03.142438 (+ 28us) server_negotiation.cc:299] Negotiation successful
0507 02:07:03.142475 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":85,"threads_started":1}
I20240507 02:07:03.143045 4160 tablet_service.cc:1467] Processing CreateTablet for tablet ba4ef5c26e47432886c6ed9b633cf678 (DEFAULT_TABLE table=test-table [id=92ca14b79fa04b909f1738dfd928ed7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:07:03.143134 4375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:03.139623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.67:43513 (local address 127.0.0.1:45260)
0507 02:07:03.140009 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:03.140019 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:03.140108 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:03.140297 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:03.140298 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:03.140300 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:07:03.140574 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:03.140575 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:03.141901 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:03.141902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:03.142935 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:03.142938 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:03.143004 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:03.143008 (+ 4us) client_negotiation.cc:769] Sending connection context
0507 02:07:03.143036 (+ 28us) client_negotiation.cc:240] Negotiation successful
0507 02:07:03.143069 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":56,"threads_started":1}
I20240507 02:07:03.143357 4160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba4ef5c26e47432886c6ed9b633cf678. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:07:03.143682 4376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:03.139710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45260 (local address 127.31.131.67:43513)
0507 02:07:03.139998 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:03.140005 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:03.140080 (+ 75us) server_negotiation.cc:407] Connection header received
0507 02:07:03.140176 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:03.140178 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:03.140193 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:03.140270 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:07:03.140641 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:03.141810 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:03.143025 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:03.143502 (+ 477us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:03.143546 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:03.143576 (+ 30us) server_negotiation.cc:299] Negotiation successful
0507 02:07:03.143610 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":114,"threads_started":1}
I20240507 02:07:03.143719 4377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:03.139859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55468 (local address 127.31.131.65:44405)
0507 02:07:03.140016 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:03.140025 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:03.140105 (+ 80us) server_negotiation.cc:407] Connection header received
0507 02:07:03.140380 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:03.140382 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:03.140398 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:03.140477 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:07:03.140756 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:03.141855 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:03.143197 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:03.143580 (+ 383us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:03.143614 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:03.143639 (+ 25us) server_negotiation.cc:299] Negotiation successful
0507 02:07:03.143665 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":89,"threads_started":1}
I20240507 02:07:03.143885 4374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:03.139748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.65:44405 (local address 127.0.0.1:55468)
0507 02:07:03.140049 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:03.140058 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:03.140113 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:03.140505 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:03.140506 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:03.140508 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:07:03.140689 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:03.140690 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:03.141929 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:03.141930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:03.143024 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:03.143026 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:03.143076 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:03.143078 (+ 2us) client_negotiation.cc:769] Sending connection context
0507 02:07:03.143105 (+ 27us) client_negotiation.cc:240] Negotiation successful
0507 02:07:03.143131 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":62,"threads_started":1}
I20240507 02:07:03.144217 4298 tablet_service.cc:1467] Processing CreateTablet for tablet ba4ef5c26e47432886c6ed9b633cf678 (DEFAULT_TABLE table=test-table [id=92ca14b79fa04b909f1738dfd928ed7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:07:03.144376 4030 tablet_service.cc:1467] Processing CreateTablet for tablet ba4ef5c26e47432886c6ed9b633cf678 (DEFAULT_TABLE table=test-table [id=92ca14b79fa04b909f1738dfd928ed7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:07:03.144469 4298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba4ef5c26e47432886c6ed9b633cf678. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:07:03.144580 4030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba4ef5c26e47432886c6ed9b633cf678. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:07:03.146250 4378 tablet_bootstrap.cc:492] T ba4ef5c26e47432886c6ed9b633cf678 P 054cfd1fcdb3401785582993601b9e0b: Bootstrap starting.
I20240507 02:07:03.147034 4379 tablet_bootstrap.cc:492] T ba4ef5c26e47432886c6ed9b633cf678 P a83ba02744de47499ab6259aa3f8c3a1: Bootstrap starting.
I20240507 02:07:03.147114 4378 tablet_bootstrap.cc:654] T ba4ef5c26e47432886c6ed9b633cf678 P 054cfd1fcdb3401785582993601b9e0b: Neither blocks nor log segments found. Creating new log.
I20240507 02:07:03.147290 4380 tablet_bootstrap.cc:492] T ba4ef5c26e47432886c6ed9b633cf678 P c0c62a5b42404d9aa35dff74b432f3a0: Bootstrap starting.
I20240507 02:07:03.147524 4378 log.cc:824] T ba4ef5c26e47432886c6ed9b633cf678 P 054cfd1fcdb3401785582993601b9e0b: Log is configured to *not* fsync() on all Append() calls
I20240507 02:07:03.147575 4379 tablet_bootstrap.cc:654] T ba4ef5c26e47432886c6ed9b633cf678 P a83ba02744de47499ab6259aa3f8c3a1: Neither blocks nor log segments found. Creating new log.
I20240507 02:07:03.147822 4380 tablet_bootstrap.cc:654] T ba4ef5c26e47432886c6ed9b633cf678 P c0c62a5b42404d9aa35dff74b432f3a0: Neither blocks nor log segments found. Creating new log.
I20240507 02:07:03.147838 4379 log.cc:824] T ba4ef5c26e47432886c6ed9b633cf678 P a83ba02744de47499ab6259aa3f8c3a1: Log is configured to *not* fsync() on all Append() calls
I20240507 02:07:03.148108 4380 log.cc:824] T ba4ef5c26e47432886c6ed9b633cf678 P c0c62a5b42404d9aa35dff74b432f3a0: Log is configured to *not* fsync() on all Append() calls
I20240507 02:07:03.148196 4378 tablet_bootstrap.cc:492] T ba4ef5c26e47432886c6ed9b633cf678 P 054cfd1fcdb3401785582993601b9e0b: No bootstrap required, opened a new log
I20240507 02:07:03.148260 4378 ts_tablet_manager.cc:1367] T ba4ef5c26e47432886c6ed9b633cf678 P 054cfd1fcdb3401785582993601b9e0b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240507 02:07:03.148458 4379 tablet_bootstrap.cc:492] T ba4ef5c26e47432886c6ed9b633cf678 P a83ba02744de47499ab6259aa3f8c3a1: No bootstrap required, opened a new log
I20240507 02:07:03.148514 4379 ts_tablet_manager.cc:1367] T ba4ef5c26e47432886c6ed9b633cf678 P a83ba02744de47499ab6259aa3f8c3a1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240507 02:07:03.148828 4380 tablet_bootstrap.cc:492] T ba4ef5c26e47432886c6ed9b633cf678 P c0c62a5b42404d9aa35dff74b432f3a0: No bootstrap required, opened a new log
I20240507 02:07:03.148928 4380 ts_tablet_manager.cc:1367] T ba4ef5c26e47432886c6ed9b633cf678 P c0c62a5b42404d9aa35dff74b432f3a0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240507 02:07:03.149998 4378 raft_consensus.cc:348] T ba4ef5c26e47432886c6ed9b633cf678 P 054cfd1fcdb3401785582993601b9e0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "054cfd1fcdb3401785582993601b9e0b" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 38921 } } peers { permanent_uuid: "c0c62a5b42404d9aa35dff74b432f3a0" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 43513 } } peers { permanent_uuid: "a83ba02744de47499ab6259aa3f8c3a1" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 44405 } }
I20240507 02:07:03.150113 4378 raft_consensus.cc:374] T ba4ef5c26e47432886c6ed9b633cf678 P 054cfd1fcdb3401785582993601b9e0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:07:03.150135 4378 raft_consensus.cc:729] T ba4ef5c26e47432886c6ed9b633cf678 P 054cfd1fcdb3401785582993601b9e0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 054cfd1fcdb3401785582993601b9e0b, State: Initialized, Role: FOLLOWER
I20240507 02:07:03.150228 4378 consensus_queue.cc:260] T ba4ef5c26e47432886c6ed9b633cf678 P 054cfd1fcdb3401785582993601b9e0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "054cfd1fcdb3401785582993601b9e0b" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 38921 } } peers { permanent_uuid: "c0c62a5b42404d9aa35dff74b432f3a0" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 43513 } } peers { permanent_uuid: "a83ba02744de47499ab6259aa3f8c3a1" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 44405 } }
I20240507 02:07:03.150444 4378 ts_tablet_manager.cc:1398] T ba4ef5c26e47432886c6ed9b633cf678 P 054cfd1fcdb3401785582993601b9e0b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240507 02:07:03.150481 4230 heartbeater.cc:503] Master 127.31.131.126:44301 was elected leader, sending a full tablet report...
I20240507 02:07:03.151325 4380 raft_consensus.cc:348] T ba4ef5c26e47432886c6ed9b633cf678 P c0c62a5b42404d9aa35dff74b432f3a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "054cfd1fcdb3401785582993601b9e0b" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 38921 } } peers { permanent_uuid: "c0c62a5b42404d9aa35dff74b432f3a0" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 43513 } } peers { permanent_uuid: "a83ba02744de47499ab6259aa3f8c3a1" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 44405 } }
I20240507 02:07:03.151460 4380 raft_consensus.cc:374] T ba4ef5c26e47432886c6ed9b633cf678 P c0c62a5b42404d9aa35dff74b432f3a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:07:03.151437 4379 raft_consensus.cc:348] T ba4ef5c26e47432886c6ed9b633cf678 P a83ba02744de47499ab6259aa3f8c3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "054cfd1fcdb3401785582993601b9e0b" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 38921 } } peers { permanent_uuid: "c0c62a5b42404d9aa35dff74b432f3a0" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 43513 } } peers { permanent_uuid: "a83ba02744de47499ab6259aa3f8c3a1" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 44405 } }
I20240507 02:07:03.151538 4380 raft_consensus.cc:729] T ba4ef5c26e47432886c6ed9b633cf678 P c0c62a5b42404d9aa35dff74b432f3a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0c62a5b42404d9aa35dff74b432f3a0, State: Initialized, Role: FOLLOWER
I20240507 02:07:03.151540 4379 raft_consensus.cc:374] T ba4ef5c26e47432886c6ed9b633cf678 P a83ba02744de47499ab6259aa3f8c3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:07:03.151628 4379 raft_consensus.cc:729] T ba4ef5c26e47432886c6ed9b633cf678 P a83ba02744de47499ab6259aa3f8c3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a83ba02744de47499ab6259aa3f8c3a1, State: Initialized, Role: FOLLOWER
I20240507 02:07:03.151672 4380 consensus_queue.cc:260] T ba4ef5c26e47432886c6ed9b633cf678 P c0c62a5b42404d9aa35dff74b432f3a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "054cfd1fcdb3401785582993601b9e0b" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 38921 } } peers { permanent_uuid: "c0c62a5b42404d9aa35dff74b432f3a0" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 43513 } } peers { permanent_uuid: "a83ba02744de47499ab6259aa3f8c3a1" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 44405 } }
I20240507 02:07:03.151715 4379 consensus_queue.cc:260] T ba4ef5c26e47432886c6ed9b633cf678 P a83ba02744de47499ab6259aa3f8c3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "054cfd1fcdb3401785582993601b9e0b" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 38921 } } peers { permanent_uuid: "c0c62a5b42404d9aa35dff74b432f3a0" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 43513 } } peers { permanent_uuid: "a83ba02744de47499ab6259aa3f8c3a1" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 44405 } }
I20240507 02:07:03.151930 4380 ts_tablet_manager.cc:1398] T ba4ef5c26e47432886c6ed9b633cf678 P c0c62a5b42404d9aa35dff74b432f3a0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240507 02:07:03.151957 4096 heartbeater.cc:503] Master 127.31.131.126:44301 was elected leader, sending a full tablet report...
I20240507 02:07:03.151957 4364 heartbeater.cc:503] Master 127.31.131.126:44301 was elected leader, sending a full tablet report...
I20240507 02:07:03.152102 4379 ts_tablet_manager.cc:1398] T ba4ef5c26e47432886c6ed9b633cf678 P a83ba02744de47499ab6259aa3f8c3a1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20240507 02:07:03.215884 4231 tablet.cc:2369] T ba4ef5c26e47432886c6ed9b633cf678 P 054cfd1fcdb3401785582993601b9e0b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240507 02:07:03.265308 4386 raft_consensus.cc:482] T ba4ef5c26e47432886c6ed9b633cf678 P a83ba02744de47499ab6259aa3f8c3a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 02:07:03.265467 4386 raft_consensus.cc:504] T ba4ef5c26e47432886c6ed9b633cf678 P a83ba02744de47499ab6259aa3f8c3a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "054cfd1fcdb3401785582993601b9e0b" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 38921 } } peers { permanent_uuid: "c0c62a5b42404d9aa35dff74b432f3a0" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 43513 } } peers { permanent_uuid: "a83ba02744de47499ab6259aa3f8c3a1" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 44405 } }
I20240507 02:07:03.265827 4386 leader_election.cc:288] T ba4ef5c26e47432886c6ed9b633cf678 P a83ba02744de47499ab6259aa3f8c3a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 054cfd1fcdb3401785582993601b9e0b (127.31.131.66:38921), c0c62a5b42404d9aa35dff74b432f3a0 (127.31.131.67:43513)
I20240507 02:07:03.268013 4098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:03.265979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.66:38921 (local address 127.31.131.65:49993)
0507 02:07:03.266036 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:03.266055 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:03.266123 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:03.266301 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:03.266303 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:03.266305 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:07:03.266530 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:03.266532 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:03.267266 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:03.267267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:03.267847 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:03.267848 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:03.267902 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:03.267906 (+ 4us) client_negotiation.cc:769] Sending connection context
0507 02:07:03.267922 (+ 16us) client_negotiation.cc:240] Negotiation successful
0507 02:07:03.267946 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20240507 02:07:03.268141 4387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:03.266076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.67:43513 (local address 127.31.131.65:48377)
0507 02:07:03.266255 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:03.266263 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:03.266318 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:03.266447 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:03.266448 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:03.266449 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:07:03.266695 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:03.266712 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:03.267332 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:03.267333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:03.267917 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:03.267919 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:03.267989 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:03.267993 (+ 4us) client_negotiation.cc:769] Sending connection context
0507 02:07:03.268028 (+ 35us) client_negotiation.cc:240] Negotiation successful
0507 02:07:03.268061 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":116,"threads_started":1}
I20240507 02:07:03.268473 4373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:03.266069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:49993 (local address 127.31.131.66:38921)
0507 02:07:03.266120 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:03.266123 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:03.266134 (+ 11us) server_negotiation.cc:407] Connection header received
0507 02:07:03.266186 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:03.266187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:03.266207 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:03.266268 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:07:03.266633 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:03.267169 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:03.267949 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:03.268374 (+ 425us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:03.268381 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:03.268403 (+ 22us) server_negotiation.cc:299] Negotiation successful
0507 02:07:03.268426 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20240507 02:07:03.268515 4376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:03.266155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:48377 (local address 127.31.131.67:43513)
0507 02:07:03.266199 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:03.266201 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:03.266323 (+ 122us) server_negotiation.cc:407] Connection header received
0507 02:07:03.266373 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:03.266374 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:03.266383 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:03.266438 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:07:03.266775 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:03.267239 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:03.268011 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:03.268425 (+ 414us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:03.268432 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:03.268448 (+ 16us) server_negotiation.cc:299] Negotiation successful
0507 02:07:03.268470 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20240507 02:07:03.268776 4318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba4ef5c26e47432886c6ed9b633cf678" candidate_uuid: "a83ba02744de47499ab6259aa3f8c3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0c62a5b42404d9aa35dff74b432f3a0" is_pre_election: true
I20240507 02:07:03.268777 4184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba4ef5c26e47432886c6ed9b633cf678" candidate_uuid: "a83ba02744de47499ab6259aa3f8c3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "054cfd1fcdb3401785582993601b9e0b" is_pre_election: true
I20240507 02:07:03.268918 4184 raft_consensus.cc:2454] T ba4ef5c26e47432886c6ed9b633cf678 P 054cfd1fcdb3401785582993601b9e0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a83ba02744de47499ab6259aa3f8c3a1 in term 0.
I20240507 02:07:03.268918 4318 raft_consensus.cc:2454] T ba4ef5c26e47432886c6ed9b633cf678 P c0c62a5b42404d9aa35dff74b432f3a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a83ba02744de47499ab6259aa3f8c3a1 in term 0.
I20240507 02:07:03.269112 3982 leader_election.cc:302] T ba4ef5c26e47432886c6ed9b633cf678 P a83ba02744de47499ab6259aa3f8c3a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a83ba02744de47499ab6259aa3f8c3a1, c0c62a5b42404d9aa35dff74b432f3a0; no voters:
I20240507 02:07:03.269248 4386 raft_consensus.cc:2781] T ba4ef5c26e47432886c6ed9b633cf678 P a83ba02744de47499ab6259aa3f8c3a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 02:07:03.269325 4386 raft_consensus.cc:482] T ba4ef5c26e47432886c6ed9b633cf678 P a83ba02744de47499ab6259aa3f8c3a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 02:07:03.269352 4386 raft_consensus.cc:3037] T ba4ef5c26e47432886c6ed9b633cf678 P a83ba02744de47499ab6259aa3f8c3a1 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:07:03.269970 4386 raft_consensus.cc:504] T ba4ef5c26e47432886c6ed9b633cf678 P a83ba02744de47499ab6259aa3f8c3a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "054cfd1fcdb3401785582993601b9e0b" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 38921 } } peers { permanent_uuid: "c0c62a5b42404d9aa35dff74b432f3a0" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 43513 } } peers { permanent_uuid: "a83ba02744de47499ab6259aa3f8c3a1" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 44405 } }
I20240507 02:07:03.270083 4386 leader_election.cc:288] T ba4ef5c26e47432886c6ed9b633cf678 P a83ba02744de47499ab6259aa3f8c3a1 [CANDIDATE]: Term 1 election: Requested vote from peers 054cfd1fcdb3401785582993601b9e0b (127.31.131.66:38921), c0c62a5b42404d9aa35dff74b432f3a0 (127.31.131.67:43513)
I20240507 02:07:03.270260 4318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba4ef5c26e47432886c6ed9b633cf678" candidate_uuid: "a83ba02744de47499ab6259aa3f8c3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0c62a5b42404d9aa35dff74b432f3a0"
I20240507 02:07:03.270258 4184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba4ef5c26e47432886c6ed9b633cf678" candidate_uuid: "a83ba02744de47499ab6259aa3f8c3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "054cfd1fcdb3401785582993601b9e0b"
I20240507 02:07:03.270349 4318 raft_consensus.cc:3037] T ba4ef5c26e47432886c6ed9b633cf678 P c0c62a5b42404d9aa35dff74b432f3a0 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:07:03.270349 4184 raft_consensus.cc:3037] T ba4ef5c26e47432886c6ed9b633cf678 P 054cfd1fcdb3401785582993601b9e0b [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:07:03.271281 4318 raft_consensus.cc:2454] T ba4ef5c26e47432886c6ed9b633cf678 P c0c62a5b42404d9aa35dff74b432f3a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a83ba02744de47499ab6259aa3f8c3a1 in term 1.
I20240507 02:07:03.271281 4184 raft_consensus.cc:2454] T ba4ef5c26e47432886c6ed9b633cf678 P 054cfd1fcdb3401785582993601b9e0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a83ba02744de47499ab6259aa3f8c3a1 in term 1.
I20240507 02:07:03.271445 3982 leader_election.cc:302] T ba4ef5c26e47432886c6ed9b633cf678 P a83ba02744de47499ab6259aa3f8c3a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a83ba02744de47499ab6259aa3f8c3a1, c0c62a5b42404d9aa35dff74b432f3a0; no voters:
I20240507 02:07:03.271543 4386 raft_consensus.cc:2781] T ba4ef5c26e47432886c6ed9b633cf678 P a83ba02744de47499ab6259aa3f8c3a1 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:07:03.271696 4386 raft_consensus.cc:686] T ba4ef5c26e47432886c6ed9b633cf678 P a83ba02744de47499ab6259aa3f8c3a1 [term 1 LEADER]: Becoming Leader. State: Replica: a83ba02744de47499ab6259aa3f8c3a1, State: Running, Role: LEADER
I20240507 02:07:03.271791 4386 consensus_queue.cc:237] T ba4ef5c26e47432886c6ed9b633cf678 P a83ba02744de47499ab6259aa3f8c3a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "054cfd1fcdb3401785582993601b9e0b" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 38921 } } peers { permanent_uuid: "c0c62a5b42404d9aa35dff74b432f3a0" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 43513 } } peers { permanent_uuid: "a83ba02744de47499ab6259aa3f8c3a1" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 44405 } }
I20240507 02:07:03.272572 3903 catalog_manager.cc:5484] T ba4ef5c26e47432886c6ed9b633cf678 P a83ba02744de47499ab6259aa3f8c3a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to a83ba02744de47499ab6259aa3f8c3a1 (127.31.131.65). New cstate: current_term: 1 leader_uuid: "a83ba02744de47499ab6259aa3f8c3a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "054cfd1fcdb3401785582993601b9e0b" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 38921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0c62a5b42404d9aa35dff74b432f3a0" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 43513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a83ba02744de47499ab6259aa3f8c3a1" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 44405 } health_report { overall_health: HEALTHY } } }
I20240507 02:07:03.288102 4377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:03.286402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55482 (local address 127.31.131.65:44405)
0507 02:07:03.286451 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:03.286453 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:03.286459 (+ 6us) server_negotiation.cc:407] Connection header received
0507 02:07:03.286494 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:03.286495 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:03.286511 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:03.286622 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0507 02:07:03.286819 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:03.287261 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:03.287732 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:03.287837 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:03.287866 (+ 29us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0507 02:07:03.287943 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0507 02:07:03.287994 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:03.288037 (+ 43us) server_negotiation.cc:299] Negotiation successful
0507 02:07:03.288053 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20240507 02:07:03.289562 4318 raft_consensus.cc:1261] T ba4ef5c26e47432886c6ed9b633cf678 P c0c62a5b42404d9aa35dff74b432f3a0 [term 1 FOLLOWER]: Refusing update from remote peer a83ba02744de47499ab6259aa3f8c3a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 02:07:03.289562 4184 raft_consensus.cc:1261] T ba4ef5c26e47432886c6ed9b633cf678 P 054cfd1fcdb3401785582993601b9e0b [term 1 FOLLOWER]: Refusing update from remote peer a83ba02744de47499ab6259aa3f8c3a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 02:07:03.289901 4386 consensus_queue.cc:1035] T ba4ef5c26e47432886c6ed9b633cf678 P a83ba02744de47499ab6259aa3f8c3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "054cfd1fcdb3401785582993601b9e0b" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 38921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 02:07:03.290032 4388 consensus_queue.cc:1035] T ba4ef5c26e47432886c6ed9b633cf678 P a83ba02744de47499ab6259aa3f8c3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0c62a5b42404d9aa35dff74b432f3a0" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 43513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 02:07:03.293423 4391 mvcc.cc:205] Tried to move back new op lower bound from 7024835064991694848 to 7024835064921726976. Current Snapshot: MvccSnapshot[applied={T|T < 7024835064991694848}]
I20240507 02:07:03.293412 4393 mvcc.cc:205] Tried to move back new op lower bound from 7024835064991694848 to 7024835064921726976. Current Snapshot: MvccSnapshot[applied={T|T < 7024835064991694848}]
I20240507 02:07:03.295349 3905 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42708:
table { table_name: "test-table" } modify_external_catalogs: true
I20240507 02:07:03.295440 3905 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42708:
table { table_name: "test-table" } modify_external_catalogs: true
I20240507 02:07:03.296087 3905 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d1760ccb8b9143af83a5684793dcd5ca: Sending DeleteTablet for 3 replicas of tablet ba4ef5c26e47432886c6ed9b633cf678
I20240507 02:07:03.296380 4298 tablet_service.cc:1514] Processing DeleteTablet for tablet ba4ef5c26e47432886c6ed9b633cf678 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:07:03 UTC) from {username='kudu', principal='kudu/127.31.131.126@KRBTEST.COM'} at 127.0.0.1:45260
I20240507 02:07:03.296407 4030 tablet_service.cc:1514] Processing DeleteTablet for tablet ba4ef5c26e47432886c6ed9b633cf678 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:07:03 UTC) from {username='kudu', principal='kudu/127.31.131.126@KRBTEST.COM'} at 127.0.0.1:55468
I20240507 02:07:03.296587 4401 tablet_replica.cc:330] T ba4ef5c26e47432886c6ed9b633cf678 P a83ba02744de47499ab6259aa3f8c3a1: stopping tablet replica
I20240507 02:07:03.296684 4160 tablet_service.cc:1514] Processing DeleteTablet for tablet ba4ef5c26e47432886c6ed9b633cf678 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:07:03 UTC) from {username='kudu', principal='kudu/127.31.131.126@KRBTEST.COM'} at 127.0.0.1:60732
I20240507 02:07:03.296855 4401 raft_consensus.cc:2229] T ba4ef5c26e47432886c6ed9b633cf678 P a83ba02744de47499ab6259aa3f8c3a1 [term 1 LEADER]: Raft consensus shutting down.
I20240507 02:07:03.296861 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 3967
I20240507 02:07:03.296875 4403 tablet_replica.cc:330] T ba4ef5c26e47432886c6ed9b633cf678 P 054cfd1fcdb3401785582993601b9e0b: stopping tablet replica
I20240507 02:07:03.296947 4403 raft_consensus.cc:2229] T ba4ef5c26e47432886c6ed9b633cf678 P 054cfd1fcdb3401785582993601b9e0b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 02:07:03.297052 4403 raft_consensus.cc:2258] T ba4ef5c26e47432886c6ed9b633cf678 P 054cfd1fcdb3401785582993601b9e0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 02:07:03.297050 4401 raft_consensus.cc:2258] T ba4ef5c26e47432886c6ed9b633cf678 P a83ba02744de47499ab6259aa3f8c3a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 02:07:03.297191 4402 tablet_replica.cc:330] T ba4ef5c26e47432886c6ed9b633cf678 P c0c62a5b42404d9aa35dff74b432f3a0: stopping tablet replica
I20240507 02:07:03.297415 4402 raft_consensus.cc:2229] T ba4ef5c26e47432886c6ed9b633cf678 P c0c62a5b42404d9aa35dff74b432f3a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 02:07:03.297510 4402 raft_consensus.cc:2258] T ba4ef5c26e47432886c6ed9b633cf678 P c0c62a5b42404d9aa35dff74b432f3a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 02:07:03.297742 4403 ts_tablet_manager.cc:1875] T ba4ef5c26e47432886c6ed9b633cf678 P 054cfd1fcdb3401785582993601b9e0b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 02:07:03.297899 4402 ts_tablet_manager.cc:1875] T ba4ef5c26e47432886c6ed9b633cf678 P c0c62a5b42404d9aa35dff74b432f3a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 02:07:03.299086 4403 ts_tablet_manager.cc:1888] T ba4ef5c26e47432886c6ed9b633cf678 P 054cfd1fcdb3401785582993601b9e0b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240507 02:07:03.299177 4403 log.cc:1196] T ba4ef5c26e47432886c6ed9b633cf678 P 054cfd1fcdb3401785582993601b9e0b: Deleting WAL directory at /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/minicluster-data/ts-1/wal/wals/ba4ef5c26e47432886c6ed9b633cf678
I20240507 02:07:03.299275 4402 ts_tablet_manager.cc:1888] T ba4ef5c26e47432886c6ed9b633cf678 P c0c62a5b42404d9aa35dff74b432f3a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240507 02:07:03.299360 4402 log.cc:1196] T ba4ef5c26e47432886c6ed9b633cf678 P c0c62a5b42404d9aa35dff74b432f3a0: Deleting WAL directory at /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1715047604443657-32269-0/minicluster-data/ts-2/wal/wals/ba4ef5c26e47432886c6ed9b633cf678
I20240507 02:07:03.299456 4403 ts_tablet_manager.cc:1909] T ba4ef5c26e47432886c6ed9b633cf678 P 054cfd1fcdb3401785582993601b9e0b: Deleting consensus metadata
I20240507 02:07:03.299612 4402 ts_tablet_manager.cc:1909] T ba4ef5c26e47432886c6ed9b633cf678 P c0c62a5b42404d9aa35dff74b432f3a0: Deleting consensus metadata
I20240507 02:07:03.299872 3889 catalog_manager.cc:4830] TS 054cfd1fcdb3401785582993601b9e0b (127.31.131.66:38921): tablet ba4ef5c26e47432886c6ed9b633cf678 (table test-table [id=92ca14b79fa04b909f1738dfd928ed7c]) successfully deleted
I20240507 02:07:03.299968 3889 catalog_manager.cc:4830] TS c0c62a5b42404d9aa35dff74b432f3a0 (127.31.131.67:43513): tablet ba4ef5c26e47432886c6ed9b633cf678 (table test-table [id=92ca14b79fa04b909f1738dfd928ed7c]) successfully deleted
W20240507 02:07:03.302121 3889 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.131.65:44405 (error 108)
I20240507 02:07:03.302577 4374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:03.302415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.65:44405 (local address 127.0.0.1:55492)
0507 02:07:03.302476 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:03.302517 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.65:44405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20240507 02:07:03.302645 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 4101
W20240507 02:07:03.302675 3889 catalog_manager.cc:4573] TS a83ba02744de47499ab6259aa3f8c3a1 (127.31.131.65:44405): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ba4ef5c26e47432886c6ed9b633cf678: Network error: Client connection negotiation failed: client connection to 127.31.131.65:44405: connect: Connection refused (error 111)
I20240507 02:07:03.307042 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 4235
I20240507 02:07:03.310384 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 3873
2024-05-07T02:07:03Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1754 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2024-05-07T02:07:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-07T02:07:03Z Disabled control of system clock
I20240507 02:07:03.333797 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715047604443657-32269-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715047604443657-32269-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715047604443657-32269-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:39577
--webserver_interface=127.31.131.126
--webserver_port=0
--builtin_ntp_servers=127.31.131.84:46051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.131.126:39577
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20240507 02:07:03.407987 4408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:07:03.408147 4408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:07:03.408166 4408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:07:03.409404 4408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240507 02:07:03.409444 4408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:07:03.409456 4408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:07:03.409466 4408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240507 02:07:03.409489 4408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240507 02:07:03.410899 4408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:46051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715047604443657-32269-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715047604443657-32269-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.131.126:39577
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:39577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.131.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715047604443657-32269-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:07:03.411087 4408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:07:03.411310 4408 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:07:03.414820 4414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:07:03.414788 4416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:07:03.414925 4408 server_base.cc:1034] running on GCE node
W20240507 02:07:03.414800 4413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:07:03.415154 4408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:07:03.415342 4408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:07:03.416461 4408 hybrid_clock.cc:648] HybridClock initialized: now 1715047623416451 us; error 29 us; skew 500 ppm
I20240507 02:07:03.417493 4408 webserver.cc:460] Webserver started at http://127.31.131.126:39777/ using document root <none> and password file <none>
I20240507 02:07:03.417640 4408 fs_manager.cc:359] Metadata directory not provided
I20240507 02:07:03.417673 4408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:07:03.417768 4408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:07:03.418676 4408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715047604443657-32269-0/minicluster-data/master-0/data/instance:
uuid: "2d57e605f80e4116856bf01a0e0c17c7"
format_stamp: "Formatted at 2024-05-07 02:07:03 on dist-test-slave-v0bg"
server_key: "3e67c2c559a7a5395e8d04b751e8e601"
server_key_iv: "5af01594507aeb4345c6baba9e626a65"
server_key_version: "encryptionkey@0"
I20240507 02:07:03.418931 4408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715047604443657-32269-0/minicluster-data/master-0/wal/instance:
uuid: "2d57e605f80e4116856bf01a0e0c17c7"
format_stamp: "Formatted at 2024-05-07 02:07:03 on dist-test-slave-v0bg"
server_key: "3e67c2c559a7a5395e8d04b751e8e601"
server_key_iv: "5af01594507aeb4345c6baba9e626a65"
server_key_version: "encryptionkey@0"
I20240507 02:07:03.419888 4408 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240507 02:07:03.420430 4422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:03.420588 4408 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240507 02:07:03.420656 4408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715047604443657-32269-0/minicluster-data/master-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715047604443657-32269-0/minicluster-data/master-0/wal
uuid: "2d57e605f80e4116856bf01a0e0c17c7"
format_stamp: "Formatted at 2024-05-07 02:07:03 on dist-test-slave-v0bg"
server_key: "3e67c2c559a7a5395e8d04b751e8e601"
server_key_iv: "5af01594507aeb4345c6baba9e626a65"
server_key_version: "encryptionkey@0"
I20240507 02:07:03.420702 4408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715047604443657-32269-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715047604443657-32269-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715047604443657-32269-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:07:03.423900 4408 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:07:03.424110 4408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:07:03.424180 4408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:07:03.427470 4408 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.126:39577
I20240507 02:07:03.427526 4475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.126:39577 every 8 connection(s)
I20240507 02:07:03.427785 4408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
I20240507 02:07:03.428270 4476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:07:03.430389 4476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d57e605f80e4116856bf01a0e0c17c7: Bootstrap starting.
I20240507 02:07:03.430904 4476 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d57e605f80e4116856bf01a0e0c17c7: Neither blocks nor log segments found. Creating new log.
I20240507 02:07:03.431131 4476 log.cc:824] T 00000000000000000000000000000000 P 2d57e605f80e4116856bf01a0e0c17c7: Log is configured to *not* fsync() on all Append() calls
I20240507 02:07:03.431681 4476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d57e605f80e4116856bf01a0e0c17c7: No bootstrap required, opened a new log
I20240507 02:07:03.433001 4476 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d57e605f80e4116856bf01a0e0c17c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d57e605f80e4116856bf01a0e0c17c7" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 39577 } }
I20240507 02:07:03.433107 4476 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d57e605f80e4116856bf01a0e0c17c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:07:03.433123 4476 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d57e605f80e4116856bf01a0e0c17c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d57e605f80e4116856bf01a0e0c17c7, State: Initialized, Role: FOLLOWER
I20240507 02:07:03.433234 4476 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d57e605f80e4116856bf01a0e0c17c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d57e605f80e4116856bf01a0e0c17c7" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 39577 } }
I20240507 02:07:03.433286 4476 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d57e605f80e4116856bf01a0e0c17c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 02:07:03.433316 4476 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d57e605f80e4116856bf01a0e0c17c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 02:07:03.433362 4476 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d57e605f80e4116856bf01a0e0c17c7 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:07:03.433854 4476 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d57e605f80e4116856bf01a0e0c17c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d57e605f80e4116856bf01a0e0c17c7" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 39577 } }
I20240507 02:07:03.433935 4476 leader_election.cc:302] T 00000000000000000000000000000000 P 2d57e605f80e4116856bf01a0e0c17c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d57e605f80e4116856bf01a0e0c17c7; no voters:
I20240507 02:07:03.434068 4476 leader_election.cc:288] T 00000000000000000000000000000000 P 2d57e605f80e4116856bf01a0e0c17c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240507 02:07:03.434114 4479 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d57e605f80e4116856bf01a0e0c17c7 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:07:03.434239 4479 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d57e605f80e4116856bf01a0e0c17c7 [term 1 LEADER]: Becoming Leader. State: Replica: 2d57e605f80e4116856bf01a0e0c17c7, State: Running, Role: LEADER
I20240507 02:07:03.434265 4476 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d57e605f80e4116856bf01a0e0c17c7 [sys.catalog]: configured and running, proceeding with master startup.
I20240507 02:07:03.434316 4479 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d57e605f80e4116856bf01a0e0c17c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d57e605f80e4116856bf01a0e0c17c7" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 39577 } }
I20240507 02:07:03.434630 4481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d57e605f80e4116856bf01a0e0c17c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d57e605f80e4116856bf01a0e0c17c7. Latest consensus state: current_term: 1 leader_uuid: "2d57e605f80e4116856bf01a0e0c17c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d57e605f80e4116856bf01a0e0c17c7" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 39577 } } }
I20240507 02:07:03.434609 4480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d57e605f80e4116856bf01a0e0c17c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d57e605f80e4116856bf01a0e0c17c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d57e605f80e4116856bf01a0e0c17c7" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 39577 } } }
I20240507 02:07:03.434701 4481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d57e605f80e4116856bf01a0e0c17c7 [sys.catalog]: This master's current role is: LEADER
I20240507 02:07:03.434706 4480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d57e605f80e4116856bf01a0e0c17c7 [sys.catalog]: This master's current role is: LEADER
W20240507 02:07:03.436136 4494 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2d57e605f80e4116856bf01a0e0c17c7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240507 02:07:03.436174 4494 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240507 02:07:03.436236 4486 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 02:07:03.436434 4486 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240507 02:07:03.437148 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 4408
I20240507 02:07:03.437245 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715047604443657-32269-0/minicluster-data/master-0/wal/instance
I20240507 02:07:03.437409 32269 external_mini_cluster.cc:1428] Setting key 144de8ef738d8f1374a72e9d7bc2cc2b
I20240507 02:07:03.437613 4486 catalog_manager.cc:1347] Generated new cluster ID: 017375a169124ae187edf109a32fa88c
I20240507 02:07:03.437660 4486 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240507 02:07:03.444468 4499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:03.442637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45252 (local address 127.31.131.126:39577)
0507 02:07:03.442837 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:03.442844 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:03.442866 (+ 22us) server_negotiation.cc:407] Connection header received
0507 02:07:03.442966 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:03.442973 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:03.443010 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:03.443157 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:03.443853 (+ 696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:03.443856 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:03.443856 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0507 02:07:03.443859 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:03.443882 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0507 02:07:03.443889 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:03.443998 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:03.444031 (+ 33us) server_negotiation.cc:299] Negotiation successful
0507 02:07:03.444074 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":68,"threads_started":1}
I20240507 02:07:03.450814 4486 catalog_manager.cc:1370] Generated new certificate authority record
I20240507 02:07:03.450939 4486 catalog_manager.cc:1504] Loading token signing keys...
I20240507 02:07:03.461354 4486 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 2d57e605f80e4116856bf01a0e0c17c7: Generated new TSK 0
I20240507 02:07:03.461565 4486 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240507 02:07:03.467159 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715047604443657-32269-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715047604443657-32269-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715047604443657-32269-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.65:0
--local_ip_for_outbound_sockets=127.31.131.65
--webserver_interface=127.31.131.65
--webserver_port=0
--tserver_master_addrs=127.31.131.126:39577
--builtin_ntp_servers=127.31.131.84:46051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20240507 02:07:03.546849 4500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:07:03.547006 4500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:07:03.547034 4500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:07:03.548247 4500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:07:03.548285 4500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:07:03.548316 4500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.65
I20240507 02:07:03.549638 4500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:46051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715047604443657-32269-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715047604443657-32269-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.131.65
--webserver_port=0
--tserver_master_addrs=127.31.131.126:39577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.65
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715047604443657-32269-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:07:03.549840 4500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:07:03.550050 4500 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:07:03.554159 4508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:07:03.554152 4506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:07:03.554297 4500 server_base.cc:1034] running on GCE node
W20240507 02:07:03.554152 4505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:07:03.554585 4500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:07:03.554793 4500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:07:03.555925 4500 hybrid_clock.cc:648] HybridClock initialized: now 1715047623555911 us; error 33 us; skew 500 ppm
I20240507 02:07:03.557058 4500 webserver.cc:460] Webserver started at http://127.31.131.65:39299/ using document root <none> and password file <none>
I20240507 02:07:03.557233 4500 fs_manager.cc:359] Metadata directory not provided
I20240507 02:07:03.557271 4500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:07:03.557358 4500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:07:03.558259 4500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715047604443657-32269-0/minicluster-data/ts-0/data/instance:
uuid: "1bb152cce7c442b587872ae0a8f2b42a"
format_stamp: "Formatted at 2024-05-07 02:07:03 on dist-test-slave-v0bg"
server_key: "259c71513b5e19341db7c3b5524ae7b4"
server_key_iv: "b19f52e9ec36cbc587aa95226533f5c0"
server_key_version: "encryptionkey@0"
I20240507 02:07:03.558519 4500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715047604443657-32269-0/minicluster-data/ts-0/wal/instance:
uuid: "1bb152cce7c442b587872ae0a8f2b42a"
format_stamp: "Formatted at 2024-05-07 02:07:03 on dist-test-slave-v0bg"
server_key: "259c71513b5e19341db7c3b5524ae7b4"
server_key_iv: "b19f52e9ec36cbc587aa95226533f5c0"
server_key_version: "encryptionkey@0"
I20240507 02:07:03.559552 4500 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240507 02:07:03.560134 4514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:03.560312 4500 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240507 02:07:03.560410 4500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715047604443657-32269-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715047604443657-32269-0/minicluster-data/ts-0/wal
uuid: "1bb152cce7c442b587872ae0a8f2b42a"
format_stamp: "Formatted at 2024-05-07 02:07:03 on dist-test-slave-v0bg"
server_key: "259c71513b5e19341db7c3b5524ae7b4"
server_key_iv: "b19f52e9ec36cbc587aa95226533f5c0"
server_key_version: "encryptionkey@0"
I20240507 02:07:03.560504 4500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715047604443657-32269-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715047604443657-32269-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715047604443657-32269-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:07:03.565395 4500 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:07:03.565629 4500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:07:03.565740 4500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:07:03.565940 4500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:07:03.566244 4500 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:07:03.566298 4500 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:03.566331 4500 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:07:03.566355 4500 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:03.572605 4500 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.65:39659
I20240507 02:07:03.572640 4628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.65:39659 every 8 connection(s)
I20240507 02:07:03.572919 4500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
I20240507 02:07:03.575141 4499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:03.573531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:55353 (local address 127.31.131.126:39577)
0507 02:07:03.573587 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:03.573589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:03.573776 (+ 187us) server_negotiation.cc:407] Connection header received
0507 02:07:03.574159 (+ 383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:03.574160 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:03.574167 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:03.574239 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:03.574925 (+ 686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:03.574927 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:03.574927 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0507 02:07:03.574931 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:03.574948 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0507 02:07:03.574951 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:03.575011 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:03.575056 (+ 45us) server_negotiation.cc:299] Negotiation successful
0507 02:07:03.575070 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20240507 02:07:03.575251 4631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:03.573489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:39577 (local address 127.31.131.65:55353)
0507 02:07:03.573690 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:03.573710 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:03.574084 (+ 374us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:03.574289 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:03.574292 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:03.574299 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:07:03.574813 (+ 514us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0507 02:07:03.574814 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:07:03.574828 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0507 02:07:03.574847 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:07:03.575009 (+ 162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:07:03.575011 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:07:03.575013 (+ 2us) client_negotiation.cc:769] Sending connection context
0507 02:07:03.575072 (+ 59us) client_negotiation.cc:240] Negotiation successful
0507 02:07:03.575104 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":68,"threads_started":1}
I20240507 02:07:03.575896 4629 heartbeater.cc:347] Connected to a master server at 127.31.131.126:39577
I20240507 02:07:03.575970 4629 heartbeater.cc:464] Registering TS with master...
I20240507 02:07:03.576077 4629 heartbeater.cc:511] Master 127.31.131.126:39577 requested a full tablet report, sending...
I20240507 02:07:03.576498 4439 ts_manager.cc:194] Registered new tserver with Master: 1bb152cce7c442b587872ae0a8f2b42a (127.31.131.65:39659)
I20240507 02:07:03.581768 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 4500
I20240507 02:07:03.581861 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715047604443657-32269-0/minicluster-data/ts-0/wal/instance
I20240507 02:07:03.581995 32269 external_mini_cluster.cc:1428] Setting key 0fb65b7b1174331e379de99f7860cd9e
I20240507 02:07:03.582806 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715047604443657-32269-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715047604443657-32269-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715047604443657-32269-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.66:0
--local_ip_for_outbound_sockets=127.31.131.66
--webserver_interface=127.31.131.66
--webserver_port=0
--tserver_master_addrs=127.31.131.126:39577
--builtin_ntp_servers=127.31.131.84:46051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20240507 02:07:03.657531 4632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:07:03.657688 4632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:07:03.657707 4632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:07:03.658957 4632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:07:03.658995 4632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:07:03.659018 4632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.66
I20240507 02:07:03.660379 4632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:46051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715047604443657-32269-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715047604443657-32269-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.131.66
--webserver_port=0
--tserver_master_addrs=127.31.131.126:39577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.66
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715047604443657-32269-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:07:03.660562 4632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:07:03.660770 4632 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:07:03.664438 4638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:07:03.664439 4637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:07:03.664588 4632 server_base.cc:1034] running on GCE node
W20240507 02:07:03.664676 4640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:07:03.664861 4632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:07:03.665055 4632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:07:03.666174 4632 hybrid_clock.cc:648] HybridClock initialized: now 1715047623666158 us; error 33 us; skew 500 ppm
I20240507 02:07:03.667299 4632 webserver.cc:460] Webserver started at http://127.31.131.66:35433/ using document root <none> and password file <none>
I20240507 02:07:03.667455 4632 fs_manager.cc:359] Metadata directory not provided
I20240507 02:07:03.667492 4632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:07:03.667588 4632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:07:03.668491 4632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715047604443657-32269-0/minicluster-data/ts-1/data/instance:
uuid: "c48bc336e12f4bc19b253028ab404e24"
format_stamp: "Formatted at 2024-05-07 02:07:03 on dist-test-slave-v0bg"
server_key: "320fd65892a11a964c37516664f56e1d"
server_key_iv: "8e8ce6d89d271181f88a97198c6c9f58"
server_key_version: "encryptionkey@0"
I20240507 02:07:03.668766 4632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715047604443657-32269-0/minicluster-data/ts-1/wal/instance:
uuid: "c48bc336e12f4bc19b253028ab404e24"
format_stamp: "Formatted at 2024-05-07 02:07:03 on dist-test-slave-v0bg"
server_key: "320fd65892a11a964c37516664f56e1d"
server_key_iv: "8e8ce6d89d271181f88a97198c6c9f58"
server_key_version: "encryptionkey@0"
I20240507 02:07:03.669994 4632 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240507 02:07:03.670575 4646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:03.670735 4632 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240507 02:07:03.670804 4632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715047604443657-32269-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715047604443657-32269-0/minicluster-data/ts-1/wal
uuid: "c48bc336e12f4bc19b253028ab404e24"
format_stamp: "Formatted at 2024-05-07 02:07:03 on dist-test-slave-v0bg"
server_key: "320fd65892a11a964c37516664f56e1d"
server_key_iv: "8e8ce6d89d271181f88a97198c6c9f58"
server_key_version: "encryptionkey@0"
I20240507 02:07:03.670863 4632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715047604443657-32269-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715047604443657-32269-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715047604443657-32269-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:07:03.674038 4632 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:07:03.674201 4632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:07:03.674288 4632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:07:03.674432 4632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:07:03.674702 4632 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:07:03.674739 4632 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:03.674762 4632 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:07:03.674778 4632 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:03.679383 4632 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.66:40729
I20240507 02:07:03.679440 4760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.66:40729 every 8 connection(s)
I20240507 02:07:03.679653 4632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
I20240507 02:07:03.681682 4499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:03.680229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.66:39491 (local address 127.31.131.126:39577)
0507 02:07:03.680276 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:03.680278 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:03.680457 (+ 179us) server_negotiation.cc:407] Connection header received
0507 02:07:03.680857 (+ 400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:03.680858 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:03.680863 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:03.680911 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:03.681497 (+ 586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:03.681498 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:03.681499 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0507 02:07:03.681502 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:03.681517 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0507 02:07:03.681521 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:03.681560 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:03.681618 (+ 58us) server_negotiation.cc:299] Negotiation successful
0507 02:07:03.681631 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20240507 02:07:03.681823 4763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:03.680206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:39577 (local address 127.31.131.66:39491)
0507 02:07:03.680368 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:03.680389 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:03.680782 (+ 393us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:03.680963 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:03.680966 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:03.680973 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:07:03.681391 (+ 418us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0507 02:07:03.681392 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:07:03.681411 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0507 02:07:03.681430 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:07:03.681563 (+ 133us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:07:03.681565 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:07:03.681567 (+ 2us) client_negotiation.cc:769] Sending connection context
0507 02:07:03.681629 (+ 62us) client_negotiation.cc:240] Negotiation successful
0507 02:07:03.681662 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":81,"threads_started":1}
I20240507 02:07:03.682322 4761 heartbeater.cc:347] Connected to a master server at 127.31.131.126:39577
I20240507 02:07:03.682394 4761 heartbeater.cc:464] Registering TS with master...
I20240507 02:07:03.682487 4761 heartbeater.cc:511] Master 127.31.131.126:39577 requested a full tablet report, sending...
I20240507 02:07:03.682799 4439 ts_manager.cc:194] Registered new tserver with Master: c48bc336e12f4bc19b253028ab404e24 (127.31.131.66:40729)
I20240507 02:07:03.686213 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 4632
I20240507 02:07:03.686291 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715047604443657-32269-0/minicluster-data/ts-1/wal/instance
I20240507 02:07:03.686435 32269 external_mini_cluster.cc:1428] Setting key 1825fc72b88b30bc661d7b4c4edf4437
I20240507 02:07:03.687218 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715047604443657-32269-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715047604443657-32269-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715047604443657-32269-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.67:0
--local_ip_for_outbound_sockets=127.31.131.67
--webserver_interface=127.31.131.67
--webserver_port=0
--tserver_master_addrs=127.31.131.126:39577
--builtin_ntp_servers=127.31.131.84:46051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20240507 02:07:03.761530 4764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:07:03.761682 4764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:07:03.761718 4764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:07:03.762953 4764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:07:03.762990 4764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:07:03.763020 4764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.67
I20240507 02:07:03.764389 4764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:46051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715047604443657-32269-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715047604443657-32269-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.131.67
--webserver_port=0
--tserver_master_addrs=127.31.131.126:39577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4764
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.67
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715047604443657-32269-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:07:03.764586 4764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:07:03.764806 4764 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:07:03.768293 4770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:07:03.768330 4772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:07:03.768473 4769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:07:03.768436 4764 server_base.cc:1034] running on GCE node
I20240507 02:07:03.768818 4764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:07:03.768992 4764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:07:03.770104 4764 hybrid_clock.cc:648] HybridClock initialized: now 1715047623770093 us; error 30 us; skew 500 ppm
I20240507 02:07:03.771180 4764 webserver.cc:460] Webserver started at http://127.31.131.67:45889/ using document root <none> and password file <none>
I20240507 02:07:03.771346 4764 fs_manager.cc:359] Metadata directory not provided
I20240507 02:07:03.771384 4764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:07:03.771474 4764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:07:03.772342 4764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715047604443657-32269-0/minicluster-data/ts-2/data/instance:
uuid: "ba68da1bd8fe4f428d9bb7e98a7d5970"
format_stamp: "Formatted at 2024-05-07 02:07:03 on dist-test-slave-v0bg"
server_key: "0d1b3bc6316417e83765f6eff1e8857e"
server_key_iv: "06c16eae335091719abe4df4ad5d7fd2"
server_key_version: "encryptionkey@0"
I20240507 02:07:03.772604 4764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715047604443657-32269-0/minicluster-data/ts-2/wal/instance:
uuid: "ba68da1bd8fe4f428d9bb7e98a7d5970"
format_stamp: "Formatted at 2024-05-07 02:07:03 on dist-test-slave-v0bg"
server_key: "0d1b3bc6316417e83765f6eff1e8857e"
server_key_iv: "06c16eae335091719abe4df4ad5d7fd2"
server_key_version: "encryptionkey@0"
I20240507 02:07:03.773705 4764 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240507 02:07:03.774283 4778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:03.774428 4764 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:03.774492 4764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715047604443657-32269-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715047604443657-32269-0/minicluster-data/ts-2/wal
uuid: "ba68da1bd8fe4f428d9bb7e98a7d5970"
format_stamp: "Formatted at 2024-05-07 02:07:03 on dist-test-slave-v0bg"
server_key: "0d1b3bc6316417e83765f6eff1e8857e"
server_key_iv: "06c16eae335091719abe4df4ad5d7fd2"
server_key_version: "encryptionkey@0"
I20240507 02:07:03.774536 4764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715047604443657-32269-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715047604443657-32269-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715047604443657-32269-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:07:03.777642 4764 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:07:03.777803 4764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:07:03.777873 4764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:07:03.777987 4764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:07:03.778179 4764 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:07:03.778203 4764 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:03.778226 4764 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:07:03.778242 4764 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:03.782797 4764 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.67:42435
I20240507 02:07:03.782833 4892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.67:42435 every 8 connection(s)
I20240507 02:07:03.783051 4764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
I20240507 02:07:03.785162 4499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:03.783660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.67:37625 (local address 127.31.131.126:39577)
0507 02:07:03.783751 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:03.783753 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:03.783868 (+ 115us) server_negotiation.cc:407] Connection header received
0507 02:07:03.784297 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:03.784298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:03.784308 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:03.784356 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:03.784930 (+ 574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:03.784932 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:03.784932 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0507 02:07:03.784936 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:03.784951 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0507 02:07:03.784955 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:03.785002 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:03.785078 (+ 76us) server_negotiation.cc:299] Negotiation successful
0507 02:07:03.785094 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20240507 02:07:03.785293 4895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:03.783610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:39577 (local address 127.31.131.67:37625)
0507 02:07:03.783779 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:03.783799 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:03.784190 (+ 391us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:03.784406 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:03.784410 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:03.784417 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:07:03.784817 (+ 400us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0507 02:07:03.784817 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:07:03.784831 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0507 02:07:03.784849 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:07:03.785007 (+ 158us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:07:03.785011 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:07:03.785013 (+ 2us) client_negotiation.cc:769] Sending connection context
0507 02:07:03.785097 (+ 84us) client_negotiation.cc:240] Negotiation successful
0507 02:07:03.785139 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":72,"threads_started":1}
I20240507 02:07:03.785791 4893 heartbeater.cc:347] Connected to a master server at 127.31.131.126:39577
I20240507 02:07:03.785863 4893 heartbeater.cc:464] Registering TS with master...
I20240507 02:07:03.785979 4893 heartbeater.cc:511] Master 127.31.131.126:39577 requested a full tablet report, sending...
I20240507 02:07:03.786293 4439 ts_manager.cc:194] Registered new tserver with Master: ba68da1bd8fe4f428d9bb7e98a7d5970 (127.31.131.67:42435)
I20240507 02:07:03.790483 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 4764
I20240507 02:07:03.790594 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715047604443657-32269-0/minicluster-data/ts-2/wal/instance
I20240507 02:07:03.790732 32269 external_mini_cluster.cc:1428] Setting key 273111ec1b4e3dc21d4fdcc5dbc2af54
I20240507 02:07:03.791468 32269 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240507 02:07:03.793382 4499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:03.792538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45264 (local address 127.31.131.126:39577)
0507 02:07:03.792595 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:03.792597 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:03.792662 (+ 65us) server_negotiation.cc:407] Connection header received
0507 02:07:03.792702 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:03.792703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:03.792707 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:03.792749 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:03.793208 (+ 459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:03.793209 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:03.793209 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0507 02:07:03.793221 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:03.793233 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0507 02:07:03.793236 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:03.793273 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:03.793310 (+ 37us) server_negotiation.cc:299] Negotiation successful
0507 02:07:03.793321 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20240507 02:07:03.794687 4439 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45264:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 02:07:03.796006 4439 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 02:07:03.799149 4902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:03.797688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.65:39659 (local address 127.0.0.1:34058)
0507 02:07:03.798061 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:03.798069 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:03.798122 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:03.798290 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:03.798291 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:03.798293 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:07:03.798736 (+ 443us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0507 02:07:03.798736 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:07:03.798763 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0507 02:07:03.798784 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:07:03.798994 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:07:03.798999 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:07:03.799002 (+ 3us) client_negotiation.cc:769] Sending connection context
0507 02:07:03.799070 (+ 68us) client_negotiation.cc:240] Negotiation successful
0507 02:07:03.799082 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"mutex_wait_us":3,"thread_start_us":48,"threads_started":1}
I20240507 02:07:03.799203 4903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:03.797784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34058 (local address 127.31.131.65:39659)
0507 02:07:03.798013 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:03.798020 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:03.798137 (+ 117us) server_negotiation.cc:407] Connection header received
0507 02:07:03.798165 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:03.798168 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:03.798179 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:03.798403 (+ 224us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:03.798890 (+ 487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:03.798893 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:03.798893 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0507 02:07:03.798897 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:03.798927 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0507 02:07:03.798932 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:03.798999 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:03.799093 (+ 94us) server_negotiation.cc:299] Negotiation successful
0507 02:07:03.799106 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":87,"threads_started":1}
I20240507 02:07:03.799569 4901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:03.797688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.66:40729 (local address 127.0.0.1:60404)
0507 02:07:03.797936 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:03.797960 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:03.798021 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:03.798196 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:03.798202 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:03.798209 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:07:03.799080 (+ 871us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0507 02:07:03.799081 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:07:03.799088 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0507 02:07:03.799099 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:07:03.799413 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:07:03.799414 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:07:03.799415 (+ 1us) client_negotiation.cc:769] Sending connection context
0507 02:07:03.799466 (+ 51us) client_negotiation.cc:240] Negotiation successful
0507 02:07:03.799479 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":75,"threads_started":1}
I20240507 02:07:03.799584 4904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:03.797784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60404 (local address 127.31.131.66:40729)
0507 02:07:03.797941 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:03.797948 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:03.798073 (+ 125us) server_negotiation.cc:407] Connection header received
0507 02:07:03.798100 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:03.798101 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:03.798111 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:03.798182 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:03.799181 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:03.799301 (+ 120us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:03.799302 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0507 02:07:03.799306 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:03.799346 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0507 02:07:03.799352 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:03.799410 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:03.799475 (+ 65us) server_negotiation.cc:299] Negotiation successful
0507 02:07:03.799488 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":83,"threads_started":1}
I20240507 02:07:03.799583 4905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:03.797870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.67:42435 (local address 127.0.0.1:59442)
0507 02:07:03.798189 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:03.798195 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:03.798231 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:03.798685 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:03.798686 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:03.798689 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:07:03.799065 (+ 376us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0507 02:07:03.799066 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:07:03.799072 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0507 02:07:03.799083 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:07:03.799450 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:07:03.799451 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:07:03.799451 (+ 0us) client_negotiation.cc:769] Sending connection context
0507 02:07:03.799490 (+ 39us) client_negotiation.cc:240] Negotiation successful
0507 02:07:03.799502 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":58,"threads_started":1}
I20240507 02:07:03.799734 4562 tablet_service.cc:1467] Processing CreateTablet for tablet 2cc626d8e704471baf305ff5ad3fde9e (DEFAULT_TABLE table=test-table [id=af8e5cdbcab64834ac17327606c7e0e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:07:03.799981 4906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:03.797946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59442 (local address 127.31.131.67:42435)
0507 02:07:03.798073 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:03.798079 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:03.798277 (+ 198us) server_negotiation.cc:407] Connection header received
0507 02:07:03.798300 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:03.798301 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:03.798310 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:03.798367 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:03.799150 (+ 783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:03.799156 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:03.799156 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0507 02:07:03.799160 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:03.799194 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0507 02:07:03.799200 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:03.799431 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:03.799614 (+ 183us) server_negotiation.cc:299] Negotiation successful
0507 02:07:03.799626 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":64,"threads_started":1}
I20240507 02:07:03.800019 4562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cc626d8e704471baf305ff5ad3fde9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:07:03.800123 4694 tablet_service.cc:1467] Processing CreateTablet for tablet 2cc626d8e704471baf305ff5ad3fde9e (DEFAULT_TABLE table=test-table [id=af8e5cdbcab64834ac17327606c7e0e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:07:03.800380 4694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cc626d8e704471baf305ff5ad3fde9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:07:03.800386 4826 tablet_service.cc:1467] Processing CreateTablet for tablet 2cc626d8e704471baf305ff5ad3fde9e (DEFAULT_TABLE table=test-table [id=af8e5cdbcab64834ac17327606c7e0e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:07:03.800623 4826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cc626d8e704471baf305ff5ad3fde9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:07:03.802456 4907 tablet_bootstrap.cc:492] T 2cc626d8e704471baf305ff5ad3fde9e P c48bc336e12f4bc19b253028ab404e24: Bootstrap starting.
I20240507 02:07:03.802860 4908 tablet_bootstrap.cc:492] T 2cc626d8e704471baf305ff5ad3fde9e P 1bb152cce7c442b587872ae0a8f2b42a: Bootstrap starting.
I20240507 02:07:03.803117 4907 tablet_bootstrap.cc:654] T 2cc626d8e704471baf305ff5ad3fde9e P c48bc336e12f4bc19b253028ab404e24: Neither blocks nor log segments found. Creating new log.
I20240507 02:07:03.803366 4909 tablet_bootstrap.cc:492] T 2cc626d8e704471baf305ff5ad3fde9e P ba68da1bd8fe4f428d9bb7e98a7d5970: Bootstrap starting.
I20240507 02:07:03.803388 4908 tablet_bootstrap.cc:654] T 2cc626d8e704471baf305ff5ad3fde9e P 1bb152cce7c442b587872ae0a8f2b42a: Neither blocks nor log segments found. Creating new log.
I20240507 02:07:03.803481 4907 log.cc:824] T 2cc626d8e704471baf305ff5ad3fde9e P c48bc336e12f4bc19b253028ab404e24: Log is configured to *not* fsync() on all Append() calls
I20240507 02:07:03.803622 4908 log.cc:824] T 2cc626d8e704471baf305ff5ad3fde9e P 1bb152cce7c442b587872ae0a8f2b42a: Log is configured to *not* fsync() on all Append() calls
I20240507 02:07:03.803833 4909 tablet_bootstrap.cc:654] T 2cc626d8e704471baf305ff5ad3fde9e P ba68da1bd8fe4f428d9bb7e98a7d5970: Neither blocks nor log segments found. Creating new log.
I20240507 02:07:03.804098 4909 log.cc:824] T 2cc626d8e704471baf305ff5ad3fde9e P ba68da1bd8fe4f428d9bb7e98a7d5970: Log is configured to *not* fsync() on all Append() calls
I20240507 02:07:03.804248 4908 tablet_bootstrap.cc:492] T 2cc626d8e704471baf305ff5ad3fde9e P 1bb152cce7c442b587872ae0a8f2b42a: No bootstrap required, opened a new log
I20240507 02:07:03.804255 4907 tablet_bootstrap.cc:492] T 2cc626d8e704471baf305ff5ad3fde9e P c48bc336e12f4bc19b253028ab404e24: No bootstrap required, opened a new log
I20240507 02:07:03.804311 4908 ts_tablet_manager.cc:1367] T 2cc626d8e704471baf305ff5ad3fde9e P 1bb152cce7c442b587872ae0a8f2b42a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240507 02:07:03.804311 4907 ts_tablet_manager.cc:1367] T 2cc626d8e704471baf305ff5ad3fde9e P c48bc336e12f4bc19b253028ab404e24: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240507 02:07:03.804685 4909 tablet_bootstrap.cc:492] T 2cc626d8e704471baf305ff5ad3fde9e P ba68da1bd8fe4f428d9bb7e98a7d5970: No bootstrap required, opened a new log
I20240507 02:07:03.804744 4909 ts_tablet_manager.cc:1367] T 2cc626d8e704471baf305ff5ad3fde9e P ba68da1bd8fe4f428d9bb7e98a7d5970: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240507 02:07:03.806286 4909 raft_consensus.cc:348] T 2cc626d8e704471baf305ff5ad3fde9e P ba68da1bd8fe4f428d9bb7e98a7d5970 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bb152cce7c442b587872ae0a8f2b42a" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 39659 } } peers { permanent_uuid: "ba68da1bd8fe4f428d9bb7e98a7d5970" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 42435 } } peers { permanent_uuid: "c48bc336e12f4bc19b253028ab404e24" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 40729 } }
I20240507 02:07:03.806392 4909 raft_consensus.cc:374] T 2cc626d8e704471baf305ff5ad3fde9e P ba68da1bd8fe4f428d9bb7e98a7d5970 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:07:03.806414 4909 raft_consensus.cc:729] T 2cc626d8e704471baf305ff5ad3fde9e P ba68da1bd8fe4f428d9bb7e98a7d5970 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba68da1bd8fe4f428d9bb7e98a7d5970, State: Initialized, Role: FOLLOWER
I20240507 02:07:03.806375 4908 raft_consensus.cc:348] T 2cc626d8e704471baf305ff5ad3fde9e P 1bb152cce7c442b587872ae0a8f2b42a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bb152cce7c442b587872ae0a8f2b42a" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 39659 } } peers { permanent_uuid: "ba68da1bd8fe4f428d9bb7e98a7d5970" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 42435 } } peers { permanent_uuid: "c48bc336e12f4bc19b253028ab404e24" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 40729 } }
I20240507 02:07:03.806375 4907 raft_consensus.cc:348] T 2cc626d8e704471baf305ff5ad3fde9e P c48bc336e12f4bc19b253028ab404e24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bb152cce7c442b587872ae0a8f2b42a" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 39659 } } peers { permanent_uuid: "ba68da1bd8fe4f428d9bb7e98a7d5970" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 42435 } } peers { permanent_uuid: "c48bc336e12f4bc19b253028ab404e24" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 40729 } }
I20240507 02:07:03.806497 4908 raft_consensus.cc:374] T 2cc626d8e704471baf305ff5ad3fde9e P 1bb152cce7c442b587872ae0a8f2b42a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:07:03.806497 4907 raft_consensus.cc:374] T 2cc626d8e704471baf305ff5ad3fde9e P c48bc336e12f4bc19b253028ab404e24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:07:03.806521 4907 raft_consensus.cc:729] T 2cc626d8e704471baf305ff5ad3fde9e P c48bc336e12f4bc19b253028ab404e24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c48bc336e12f4bc19b253028ab404e24, State: Initialized, Role: FOLLOWER
I20240507 02:07:03.806521 4908 raft_consensus.cc:729] T 2cc626d8e704471baf305ff5ad3fde9e P 1bb152cce7c442b587872ae0a8f2b42a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bb152cce7c442b587872ae0a8f2b42a, State: Initialized, Role: FOLLOWER
I20240507 02:07:03.806490 4909 consensus_queue.cc:260] T 2cc626d8e704471baf305ff5ad3fde9e P ba68da1bd8fe4f428d9bb7e98a7d5970 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bb152cce7c442b587872ae0a8f2b42a" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 39659 } } peers { permanent_uuid: "ba68da1bd8fe4f428d9bb7e98a7d5970" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 42435 } } peers { permanent_uuid: "c48bc336e12f4bc19b253028ab404e24" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 40729 } }
I20240507 02:07:03.806623 4907 consensus_queue.cc:260] T 2cc626d8e704471baf305ff5ad3fde9e P c48bc336e12f4bc19b253028ab404e24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bb152cce7c442b587872ae0a8f2b42a" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 39659 } } peers { permanent_uuid: "ba68da1bd8fe4f428d9bb7e98a7d5970" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 42435 } } peers { permanent_uuid: "c48bc336e12f4bc19b253028ab404e24" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 40729 } }
I20240507 02:07:03.806623 4908 consensus_queue.cc:260] T 2cc626d8e704471baf305ff5ad3fde9e P 1bb152cce7c442b587872ae0a8f2b42a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bb152cce7c442b587872ae0a8f2b42a" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 39659 } } peers { permanent_uuid: "ba68da1bd8fe4f428d9bb7e98a7d5970" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 42435 } } peers { permanent_uuid: "c48bc336e12f4bc19b253028ab404e24" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 40729 } }
I20240507 02:07:03.806728 4909 ts_tablet_manager.cc:1398] T 2cc626d8e704471baf305ff5ad3fde9e P ba68da1bd8fe4f428d9bb7e98a7d5970: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240507 02:07:03.806836 4908 ts_tablet_manager.cc:1398] T 2cc626d8e704471baf305ff5ad3fde9e P 1bb152cce7c442b587872ae0a8f2b42a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240507 02:07:03.806836 4907 ts_tablet_manager.cc:1398] T 2cc626d8e704471baf305ff5ad3fde9e P c48bc336e12f4bc19b253028ab404e24: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240507 02:07:03.806885 4893 heartbeater.cc:503] Master 127.31.131.126:39577 was elected leader, sending a full tablet report...
I20240507 02:07:03.806982 4629 heartbeater.cc:503] Master 127.31.131.126:39577 was elected leader, sending a full tablet report...
I20240507 02:07:03.807181 4761 heartbeater.cc:503] Master 127.31.131.126:39577 was elected leader, sending a full tablet report...
W20240507 02:07:03.823398 4630 tablet.cc:2369] T 2cc626d8e704471baf305ff5ad3fde9e P 1bb152cce7c442b587872ae0a8f2b42a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240507 02:07:03.930096 4762 tablet.cc:2369] T 2cc626d8e704471baf305ff5ad3fde9e P c48bc336e12f4bc19b253028ab404e24: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240507 02:07:03.930439 4914 raft_consensus.cc:482] T 2cc626d8e704471baf305ff5ad3fde9e P c48bc336e12f4bc19b253028ab404e24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 02:07:03.930581 4914 raft_consensus.cc:504] T 2cc626d8e704471baf305ff5ad3fde9e P c48bc336e12f4bc19b253028ab404e24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bb152cce7c442b587872ae0a8f2b42a" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 39659 } } peers { permanent_uuid: "ba68da1bd8fe4f428d9bb7e98a7d5970" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 42435 } } peers { permanent_uuid: "c48bc336e12f4bc19b253028ab404e24" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 40729 } }
I20240507 02:07:03.930888 4914 leader_election.cc:288] T 2cc626d8e704471baf305ff5ad3fde9e P c48bc336e12f4bc19b253028ab404e24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1bb152cce7c442b587872ae0a8f2b42a (127.31.131.65:39659), ba68da1bd8fe4f428d9bb7e98a7d5970 (127.31.131.67:42435)
I20240507 02:07:03.932126 4763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:03.931059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.65:39659 (local address 127.31.131.66:60495)
0507 02:07:03.931113 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:03.931128 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:03.931183 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:03.931345 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:03.931346 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:03.931349 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:07:03.931809 (+ 460us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0507 02:07:03.931809 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:07:03.931817 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0507 02:07:03.931828 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:07:03.931995 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:07:03.931996 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:07:03.931997 (+ 1us) client_negotiation.cc:769] Sending connection context
0507 02:07:03.932045 (+ 48us) client_negotiation.cc:240] Negotiation successful
0507 02:07:03.932058 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20240507 02:07:03.932251 4903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:03.931148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.66:60495 (local address 127.31.131.65:39659)
0507 02:07:03.931195 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:03.931197 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:03.931206 (+ 9us) server_negotiation.cc:407] Connection header received
0507 02:07:03.931233 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:03.931234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:03.931240 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:03.931303 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:03.931905 (+ 602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:03.931907 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:03.931907 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0507 02:07:03.931912 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:03.931929 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0507 02:07:03.931933 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:03.931981 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:03.932168 (+ 187us) server_negotiation.cc:299] Negotiation successful
0507 02:07:03.932186 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20240507 02:07:03.932297 4906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:03.931282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.66:56293 (local address 127.31.131.67:42435)
0507 02:07:03.931337 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:03.931339 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:03.931408 (+ 69us) server_negotiation.cc:407] Connection header received
0507 02:07:03.931448 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:03.931449 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:03.931457 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:03.931521 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:03.932055 (+ 534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:03.932056 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:03.932057 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0507 02:07:03.932060 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:03.932075 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0507 02:07:03.932078 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:03.932139 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:03.932201 (+ 62us) server_negotiation.cc:299] Negotiation successful
0507 02:07:03.932220 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20240507 02:07:03.932358 4916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:03.931166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.67:42435 (local address 127.31.131.66:56293)
0507 02:07:03.931362 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:03.931370 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:03.931409 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:03.931528 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:03.931529 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:03.931530 (+ 1us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:07:03.931973 (+ 443us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0507 02:07:03.931973 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:07:03.931979 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0507 02:07:03.931989 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:07:03.932138 (+ 149us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:07:03.932138 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:07:03.932139 (+ 1us) client_negotiation.cc:769] Sending connection context
0507 02:07:03.932261 (+ 122us) client_negotiation.cc:240] Negotiation successful
0507 02:07:03.932274 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":115,"threads_started":1}
I20240507 02:07:03.932685 4847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cc626d8e704471baf305ff5ad3fde9e" candidate_uuid: "c48bc336e12f4bc19b253028ab404e24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba68da1bd8fe4f428d9bb7e98a7d5970" is_pre_election: true
I20240507 02:07:03.932848 4847 raft_consensus.cc:2454] T 2cc626d8e704471baf305ff5ad3fde9e P ba68da1bd8fe4f428d9bb7e98a7d5970 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c48bc336e12f4bc19b253028ab404e24 in term 0.
I20240507 02:07:03.932855 4582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cc626d8e704471baf305ff5ad3fde9e" candidate_uuid: "c48bc336e12f4bc19b253028ab404e24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bb152cce7c442b587872ae0a8f2b42a" is_pre_election: true
I20240507 02:07:03.933024 4582 raft_consensus.cc:2454] T 2cc626d8e704471baf305ff5ad3fde9e P 1bb152cce7c442b587872ae0a8f2b42a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c48bc336e12f4bc19b253028ab404e24 in term 0.
I20240507 02:07:03.933048 4650 leader_election.cc:302] T 2cc626d8e704471baf305ff5ad3fde9e P c48bc336e12f4bc19b253028ab404e24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba68da1bd8fe4f428d9bb7e98a7d5970, c48bc336e12f4bc19b253028ab404e24; no voters:
I20240507 02:07:03.933166 4914 raft_consensus.cc:2781] T 2cc626d8e704471baf305ff5ad3fde9e P c48bc336e12f4bc19b253028ab404e24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 02:07:03.933236 4914 raft_consensus.cc:482] T 2cc626d8e704471baf305ff5ad3fde9e P c48bc336e12f4bc19b253028ab404e24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 02:07:03.933261 4914 raft_consensus.cc:3037] T 2cc626d8e704471baf305ff5ad3fde9e P c48bc336e12f4bc19b253028ab404e24 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:07:03.934053 4914 raft_consensus.cc:504] T 2cc626d8e704471baf305ff5ad3fde9e P c48bc336e12f4bc19b253028ab404e24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bb152cce7c442b587872ae0a8f2b42a" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 39659 } } peers { permanent_uuid: "ba68da1bd8fe4f428d9bb7e98a7d5970" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 42435 } } peers { permanent_uuid: "c48bc336e12f4bc19b253028ab404e24" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 40729 } }
I20240507 02:07:03.934190 4914 leader_election.cc:288] T 2cc626d8e704471baf305ff5ad3fde9e P c48bc336e12f4bc19b253028ab404e24 [CANDIDATE]: Term 1 election: Requested vote from peers 1bb152cce7c442b587872ae0a8f2b42a (127.31.131.65:39659), ba68da1bd8fe4f428d9bb7e98a7d5970 (127.31.131.67:42435)
I20240507 02:07:03.934378 4847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cc626d8e704471baf305ff5ad3fde9e" candidate_uuid: "c48bc336e12f4bc19b253028ab404e24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba68da1bd8fe4f428d9bb7e98a7d5970"
I20240507 02:07:03.934386 4582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cc626d8e704471baf305ff5ad3fde9e" candidate_uuid: "c48bc336e12f4bc19b253028ab404e24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bb152cce7c442b587872ae0a8f2b42a"
I20240507 02:07:03.934448 4847 raft_consensus.cc:3037] T 2cc626d8e704471baf305ff5ad3fde9e P ba68da1bd8fe4f428d9bb7e98a7d5970 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:07:03.934451 4582 raft_consensus.cc:3037] T 2cc626d8e704471baf305ff5ad3fde9e P 1bb152cce7c442b587872ae0a8f2b42a [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:07:03.935223 4847 raft_consensus.cc:2454] T 2cc626d8e704471baf305ff5ad3fde9e P ba68da1bd8fe4f428d9bb7e98a7d5970 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c48bc336e12f4bc19b253028ab404e24 in term 1.
I20240507 02:07:03.935257 4582 raft_consensus.cc:2454] T 2cc626d8e704471baf305ff5ad3fde9e P 1bb152cce7c442b587872ae0a8f2b42a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c48bc336e12f4bc19b253028ab404e24 in term 1.
I20240507 02:07:03.935387 4650 leader_election.cc:302] T 2cc626d8e704471baf305ff5ad3fde9e P c48bc336e12f4bc19b253028ab404e24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bb152cce7c442b587872ae0a8f2b42a, c48bc336e12f4bc19b253028ab404e24; no voters:
I20240507 02:07:03.935500 4914 raft_consensus.cc:2781] T 2cc626d8e704471baf305ff5ad3fde9e P c48bc336e12f4bc19b253028ab404e24 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:07:03.935629 4914 raft_consensus.cc:686] T 2cc626d8e704471baf305ff5ad3fde9e P c48bc336e12f4bc19b253028ab404e24 [term 1 LEADER]: Becoming Leader. State: Replica: c48bc336e12f4bc19b253028ab404e24, State: Running, Role: LEADER
I20240507 02:07:03.935726 4914 consensus_queue.cc:237] T 2cc626d8e704471baf305ff5ad3fde9e P c48bc336e12f4bc19b253028ab404e24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bb152cce7c442b587872ae0a8f2b42a" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 39659 } } peers { permanent_uuid: "ba68da1bd8fe4f428d9bb7e98a7d5970" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 42435 } } peers { permanent_uuid: "c48bc336e12f4bc19b253028ab404e24" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 40729 } }
I20240507 02:07:03.936520 4439 catalog_manager.cc:5484] T 2cc626d8e704471baf305ff5ad3fde9e P c48bc336e12f4bc19b253028ab404e24 reported cstate change: term changed from 0 to 1, leader changed from <none> to c48bc336e12f4bc19b253028ab404e24 (127.31.131.66). New cstate: current_term: 1 leader_uuid: "c48bc336e12f4bc19b253028ab404e24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bb152cce7c442b587872ae0a8f2b42a" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 39659 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba68da1bd8fe4f428d9bb7e98a7d5970" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 42435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c48bc336e12f4bc19b253028ab404e24" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 40729 } health_report { overall_health: HEALTHY } } }
I20240507 02:07:03.945214 4904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:03.944397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60418 (local address 127.31.131.66:40729)
0507 02:07:03.944447 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:03.944448 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:03.944455 (+ 7us) server_negotiation.cc:407] Connection header received
0507 02:07:03.944497 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:03.944497 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:03.944508 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:03.944546 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:03.945019 (+ 473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:03.945020 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:03.945020 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0507 02:07:03.945024 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:03.945037 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0507 02:07:03.945040 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:03.945085 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:03.945134 (+ 49us) server_negotiation.cc:299] Negotiation successful
0507 02:07:03.945145 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20240507 02:07:03.946565 4582 raft_consensus.cc:1261] T 2cc626d8e704471baf305ff5ad3fde9e P 1bb152cce7c442b587872ae0a8f2b42a [term 1 FOLLOWER]: Refusing update from remote peer c48bc336e12f4bc19b253028ab404e24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 02:07:03.946568 4847 raft_consensus.cc:1261] T 2cc626d8e704471baf305ff5ad3fde9e P ba68da1bd8fe4f428d9bb7e98a7d5970 [term 1 FOLLOWER]: Refusing update from remote peer c48bc336e12f4bc19b253028ab404e24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 02:07:03.946869 4917 consensus_queue.cc:1035] T 2cc626d8e704471baf305ff5ad3fde9e P c48bc336e12f4bc19b253028ab404e24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bb152cce7c442b587872ae0a8f2b42a" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 39659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 02:07:03.946985 4914 consensus_queue.cc:1035] T 2cc626d8e704471baf305ff5ad3fde9e P c48bc336e12f4bc19b253028ab404e24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba68da1bd8fe4f428d9bb7e98a7d5970" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 42435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 02:07:03.948885 4920 mvcc.cc:205] Tried to move back new op lower bound from 7024835067682979840 to 7024835067641364480. Current Snapshot: MvccSnapshot[applied={T|T < 7024835067682979840}]
I20240507 02:07:03.951042 4438 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45264:
table { table_name: "test-table" } modify_external_catalogs: true
I20240507 02:07:03.951114 4438 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45264:
table { table_name: "test-table" } modify_external_catalogs: true
I20240507 02:07:03.951766 4438 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 2d57e605f80e4116856bf01a0e0c17c7: Sending DeleteTablet for 3 replicas of tablet 2cc626d8e704471baf305ff5ad3fde9e
I20240507 02:07:03.952039 4694 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cc626d8e704471baf305ff5ad3fde9e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:07:03 UTC) from {username='slave'} at 127.0.0.1:60404
I20240507 02:07:03.952044 4826 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cc626d8e704471baf305ff5ad3fde9e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:07:03 UTC) from {username='slave'} at 127.0.0.1:59442
I20240507 02:07:03.952054 4562 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cc626d8e704471baf305ff5ad3fde9e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:07:03 UTC) from {username='slave'} at 127.0.0.1:34058
I20240507 02:07:03.952229 4930 tablet_replica.cc:330] T 2cc626d8e704471baf305ff5ad3fde9e P c48bc336e12f4bc19b253028ab404e24: stopping tablet replica
I20240507 02:07:03.952239 4932 tablet_replica.cc:330] T 2cc626d8e704471baf305ff5ad3fde9e P ba68da1bd8fe4f428d9bb7e98a7d5970: stopping tablet replica
I20240507 02:07:03.952302 4932 raft_consensus.cc:2229] T 2cc626d8e704471baf305ff5ad3fde9e P ba68da1bd8fe4f428d9bb7e98a7d5970 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 02:07:03.952302 4930 raft_consensus.cc:2229] T 2cc626d8e704471baf305ff5ad3fde9e P c48bc336e12f4bc19b253028ab404e24 [term 1 LEADER]: Raft consensus shutting down.
I20240507 02:07:03.952380 4930 raft_consensus.cc:2258] T 2cc626d8e704471baf305ff5ad3fde9e P c48bc336e12f4bc19b253028ab404e24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 02:07:03.952383 4932 raft_consensus.cc:2258] T 2cc626d8e704471baf305ff5ad3fde9e P ba68da1bd8fe4f428d9bb7e98a7d5970 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 02:07:03.952373 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 4500
I20240507 02:07:03.953254 4932 ts_tablet_manager.cc:1875] T 2cc626d8e704471baf305ff5ad3fde9e P ba68da1bd8fe4f428d9bb7e98a7d5970: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 02:07:03.954773 4932 ts_tablet_manager.cc:1888] T 2cc626d8e704471baf305ff5ad3fde9e P ba68da1bd8fe4f428d9bb7e98a7d5970: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240507 02:07:03.954898 4932 log.cc:1196] T 2cc626d8e704471baf305ff5ad3fde9e P ba68da1bd8fe4f428d9bb7e98a7d5970: Deleting WAL directory at /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715047604443657-32269-0/minicluster-data/ts-2/wal/wals/2cc626d8e704471baf305ff5ad3fde9e
I20240507 02:07:03.955224 4932 ts_tablet_manager.cc:1909] T 2cc626d8e704471baf305ff5ad3fde9e P ba68da1bd8fe4f428d9bb7e98a7d5970: Deleting consensus metadata
I20240507 02:07:03.955585 4930 ts_tablet_manager.cc:1875] T 2cc626d8e704471baf305ff5ad3fde9e P c48bc336e12f4bc19b253028ab404e24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 02:07:03.955641 4426 catalog_manager.cc:4830] TS ba68da1bd8fe4f428d9bb7e98a7d5970 (127.31.131.67:42435): tablet 2cc626d8e704471baf305ff5ad3fde9e (table test-table [id=af8e5cdbcab64834ac17327606c7e0e3]) successfully deleted
I20240507 02:07:03.957163 4930 ts_tablet_manager.cc:1888] T 2cc626d8e704471baf305ff5ad3fde9e P c48bc336e12f4bc19b253028ab404e24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240507 02:07:03.957235 4930 log.cc:1196] T 2cc626d8e704471baf305ff5ad3fde9e P c48bc336e12f4bc19b253028ab404e24: Deleting WAL directory at /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1715047604443657-32269-0/minicluster-data/ts-1/wal/wals/2cc626d8e704471baf305ff5ad3fde9e
I20240507 02:07:03.957440 4930 ts_tablet_manager.cc:1909] T 2cc626d8e704471baf305ff5ad3fde9e P c48bc336e12f4bc19b253028ab404e24: Deleting consensus metadata
I20240507 02:07:03.957806 4424 catalog_manager.cc:4830] TS c48bc336e12f4bc19b253028ab404e24 (127.31.131.66:40729): tablet 2cc626d8e704471baf305ff5ad3fde9e (table test-table [id=af8e5cdbcab64834ac17327606c7e0e3]) successfully deleted
W20240507 02:07:03.958251 4426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.131.65:39659 (error 108)
I20240507 02:07:03.958391 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 4632
I20240507 02:07:03.959165 4902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:03.959064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.65:39659 (local address 127.0.0.1:34066)
0507 02:07:03.959105 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:03.959134 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.65:39659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20240507 02:07:03.959281 4426 catalog_manager.cc:4573] TS 1bb152cce7c442b587872ae0a8f2b42a (127.31.131.65:39659): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2cc626d8e704471baf305ff5ad3fde9e: Network error: Client connection negotiation failed: client connection to 127.31.131.65:39659: connect: Connection refused (error 111)
I20240507 02:07:03.962502 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 4764
I20240507 02:07:03.966614 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 4408
2024-05-07T02:07:03Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (655 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20240507 02:07:03.984159 32269 mini_oidc.cc:152] Starting JWKS server
I20240507 02:07:03.985158 32269 webserver.cc:460] Webserver started at https://127.0.0.1:39141/ using document root <none> and password file <none>
I20240507 02:07:03.985236 32269 mini_oidc.cc:189] Starting OIDC Discovery server
I20240507 02:07:03.985381 32269 webserver.cc:460] Webserver started at http://127.0.0.1:38335/ using document root <none> and password file <none>
2024-05-07T02:07:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-07T02:07:03Z Disabled control of system clock
I20240507 02:07:03.991283 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1715047604443657-32269-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1715047604443657-32269-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1715047604443657-32269-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:39473
--webserver_interface=127.31.131.126
--webserver_port=0
--builtin_ntp_servers=127.31.131.84:37023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.131.126:39473
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39141/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1715047604443657-32269-0/testchainca.cert with env {}
W20240507 02:07:04.065549 4939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:07:04.065702 4939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:07:04.065722 4939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:07:04.066964 4939 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240507 02:07:04.067003 4939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:07:04.067014 4939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:07:04.067024 4939 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240507 02:07:04.067034 4939 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20240507 02:07:04.067044 4939 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20240507 02:07:04.067052 4939 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39141/jwks.json
I20240507 02:07:04.068591 4939 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:37023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1715047604443657-32269-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1715047604443657-32269-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.131.126:39473
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:39473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39141/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.131.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1715047604443657-32269-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1715047604443657-32269-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:07:04.068742 4939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:07:04.068871 4939 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:07:04.072229 4945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:07:04.072234 4947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:07:04.072306 4939 server_base.cc:1034] running on GCE node
W20240507 02:07:04.072242 4944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:07:04.072597 4939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:07:04.072820 4939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:07:04.073962 4939 hybrid_clock.cc:648] HybridClock initialized: now 1715047624073950 us; error 23 us; skew 500 ppm
I20240507 02:07:04.075038 4939 webserver.cc:460] Webserver started at http://127.31.131.126:44253/ using document root <none> and password file <none>
I20240507 02:07:04.075197 4939 fs_manager.cc:359] Metadata directory not provided
I20240507 02:07:04.075238 4939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:07:04.075336 4939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:07:04.076242 4939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1715047604443657-32269-0/minicluster-data/master-0/data/instance:
uuid: "71a6b2e953bd4520b5992d9cb0a28aa3"
format_stamp: "Formatted at 2024-05-07 02:07:04 on dist-test-slave-v0bg"
server_key: "35c914b7b1260091f6991e1de3d7fc73"
server_key_iv: "c15deccf03e303a0ca68a3cb7c254266"
server_key_version: "encryptionkey@0"
I20240507 02:07:04.076521 4939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1715047604443657-32269-0/minicluster-data/master-0/wal/instance:
uuid: "71a6b2e953bd4520b5992d9cb0a28aa3"
format_stamp: "Formatted at 2024-05-07 02:07:04 on dist-test-slave-v0bg"
server_key: "35c914b7b1260091f6991e1de3d7fc73"
server_key_iv: "c15deccf03e303a0ca68a3cb7c254266"
server_key_version: "encryptionkey@0"
I20240507 02:07:04.077530 4939 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240507 02:07:04.078089 4953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:04.078243 4939 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:04.078312 4939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1715047604443657-32269-0/minicluster-data/master-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1715047604443657-32269-0/minicluster-data/master-0/wal
uuid: "71a6b2e953bd4520b5992d9cb0a28aa3"
format_stamp: "Formatted at 2024-05-07 02:07:04 on dist-test-slave-v0bg"
server_key: "35c914b7b1260091f6991e1de3d7fc73"
server_key_iv: "c15deccf03e303a0ca68a3cb7c254266"
server_key_version: "encryptionkey@0"
I20240507 02:07:04.078366 4939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1715047604443657-32269-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1715047604443657-32269-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1715047604443657-32269-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:07:04.116875 4939 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:07:04.117180 4939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:07:04.117287 4939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:07:04.120843 4939 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.126:39473
I20240507 02:07:04.120883 5006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.126:39473 every 8 connection(s)
I20240507 02:07:04.121169 4939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
I20240507 02:07:04.121726 5007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:07:04.124140 5007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71a6b2e953bd4520b5992d9cb0a28aa3: Bootstrap starting.
I20240507 02:07:04.124531 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 4939
I20240507 02:07:04.124620 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1715047604443657-32269-0/minicluster-data/master-0/wal/instance
I20240507 02:07:04.124624 5007 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71a6b2e953bd4520b5992d9cb0a28aa3: Neither blocks nor log segments found. Creating new log.
I20240507 02:07:04.124769 32269 external_mini_cluster.cc:1428] Setting key 1fe33e9d9b0c2abbdcb33437c9fdd659
I20240507 02:07:04.124869 5007 log.cc:824] T 00000000000000000000000000000000 P 71a6b2e953bd4520b5992d9cb0a28aa3: Log is configured to *not* fsync() on all Append() calls
I20240507 02:07:04.125581 5007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71a6b2e953bd4520b5992d9cb0a28aa3: No bootstrap required, opened a new log
I20240507 02:07:04.127043 5007 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71a6b2e953bd4520b5992d9cb0a28aa3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71a6b2e953bd4520b5992d9cb0a28aa3" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 39473 } }
I20240507 02:07:04.127161 5007 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71a6b2e953bd4520b5992d9cb0a28aa3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:07:04.127182 5007 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71a6b2e953bd4520b5992d9cb0a28aa3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a6b2e953bd4520b5992d9cb0a28aa3, State: Initialized, Role: FOLLOWER
I20240507 02:07:04.127274 5007 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71a6b2e953bd4520b5992d9cb0a28aa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71a6b2e953bd4520b5992d9cb0a28aa3" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 39473 } }
I20240507 02:07:04.127349 5007 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71a6b2e953bd4520b5992d9cb0a28aa3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 02:07:04.127377 5007 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71a6b2e953bd4520b5992d9cb0a28aa3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 02:07:04.127424 5007 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71a6b2e953bd4520b5992d9cb0a28aa3 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:07:04.128036 5007 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71a6b2e953bd4520b5992d9cb0a28aa3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71a6b2e953bd4520b5992d9cb0a28aa3" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 39473 } }
I20240507 02:07:04.128427 5007 leader_election.cc:302] T 00000000000000000000000000000000 P 71a6b2e953bd4520b5992d9cb0a28aa3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71a6b2e953bd4520b5992d9cb0a28aa3; no voters:
I20240507 02:07:04.128602 5007 leader_election.cc:288] T 00000000000000000000000000000000 P 71a6b2e953bd4520b5992d9cb0a28aa3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240507 02:07:04.128821 5007 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71a6b2e953bd4520b5992d9cb0a28aa3 [sys.catalog]: configured and running, proceeding with master startup.
I20240507 02:07:04.128995 5012 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71a6b2e953bd4520b5992d9cb0a28aa3 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:07:04.129187 5012 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71a6b2e953bd4520b5992d9cb0a28aa3 [term 1 LEADER]: Becoming Leader. State: Replica: 71a6b2e953bd4520b5992d9cb0a28aa3, State: Running, Role: LEADER
I20240507 02:07:04.129297 5012 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71a6b2e953bd4520b5992d9cb0a28aa3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71a6b2e953bd4520b5992d9cb0a28aa3" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 39473 } }
I20240507 02:07:04.129336 5011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:04.125206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47886 (local address 127.31.131.126:39473)
0507 02:07:04.125413 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:04.125419 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:04.125437 (+ 18us) server_negotiation.cc:407] Connection header received
0507 02:07:04.125896 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:04.125902 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:04.125953 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:04.126069 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:04.126488 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:04.127221 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:04.127985 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:04.128172 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:04.128577 (+ 405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:04.128585 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:04.128585 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0507 02:07:04.128774 (+ 189us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:04.128809 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0507 02:07:04.128817 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:04.129061 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:04.129100 (+ 39us) server_negotiation.cc:299] Negotiation successful
0507 02:07:04.129152 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":89,"threads_started":1}
I20240507 02:07:04.129784 5014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71a6b2e953bd4520b5992d9cb0a28aa3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71a6b2e953bd4520b5992d9cb0a28aa3. Latest consensus state: current_term: 1 leader_uuid: "71a6b2e953bd4520b5992d9cb0a28aa3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71a6b2e953bd4520b5992d9cb0a28aa3" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 39473 } } }
I20240507 02:07:04.129863 5014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71a6b2e953bd4520b5992d9cb0a28aa3 [sys.catalog]: This master's current role is: LEADER
I20240507 02:07:04.130159 32269 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240507 02:07:04.130234 5013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71a6b2e953bd4520b5992d9cb0a28aa3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71a6b2e953bd4520b5992d9cb0a28aa3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71a6b2e953bd4520b5992d9cb0a28aa3" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 39473 } } }
I20240507 02:07:04.130383 5013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71a6b2e953bd4520b5992d9cb0a28aa3 [sys.catalog]: This master's current role is: LEADER
W20240507 02:07:04.130935 5027 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 71a6b2e953bd4520b5992d9cb0a28aa3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240507 02:07:04.130980 5027 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240507 02:07:04.131016 5022 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 02:07:04.131134 5022 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240507 02:07:04.132228 5022 catalog_manager.cc:1347] Generated new cluster ID: 240ee6f4bd1540ed82937e02c37e2c7c
I20240507 02:07:04.132280 5022 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240507 02:07:04.150275 5022 catalog_manager.cc:1370] Generated new certificate authority record
I20240507 02:07:04.151043 5022 catalog_manager.cc:1504] Loading token signing keys...
I20240507 02:07:04.157709 5022 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 71a6b2e953bd4520b5992d9cb0a28aa3: Generated new TSK 0
I20240507 02:07:04.157876 5022 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240507 02:07:04.182211 5011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:04.164075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47894 (local address 127.31.131.126:39473)
0507 02:07:04.164131 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:04.164133 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:04.164211 (+ 78us) server_negotiation.cc:407] Connection header received
0507 02:07:04.164295 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:04.164296 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:04.164308 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:04.164387 (+ 79us) server_negotiation.cc:226] Negotiated authn=JWT
0507 02:07:04.164636 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:04.165442 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:04.165999 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:04.166180 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:04.166232 (+ 52us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0507 02:07:04.181933 (+ 15701us) server_negotiation.cc:377] Sending JWT_EXCHANGE NegotiatePB response
0507 02:07:04.182010 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:04.182095 (+ 85us) server_negotiation.cc:299] Negotiation successful
0507 02:07:04.182123 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34,"thread_start_us":108,"threads_started":1}
I20240507 02:07:04.190024 5011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:04.187539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47910 (local address 127.31.131.126:39473)
0507 02:07:04.187579 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:04.187581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:04.187636 (+ 55us) server_negotiation.cc:407] Connection header received
0507 02:07:04.187687 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:04.187688 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:04.187697 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:04.187756 (+ 59us) server_negotiation.cc:226] Negotiated authn=JWT
0507 02:07:04.187977 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:04.188715 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:04.189231 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:04.189408 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:04.189430 (+ 22us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0507 02:07:04.189828 (+ 398us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
0507 02:07:04.189960 (+ 132us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47910: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
Metrics: {"server-negotiator.queue_time_us":19}
W20240507 02:07:04.190089 5011 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47910: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
W20240507 02:07:04.190209 5043 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:07:04.187467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:39473 (local address 127.0.0.1:47910)
0507 02:07:04.187589 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:04.187597 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:04.187648 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:04.187769 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:04.187770 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:04.187772 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0507 02:07:04.187926 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:04.187927 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:04.188793 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:04.188794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:04.189163 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:04.189165 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:04.189215 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:04.189219 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0507 02:07:04.189953 (+ 734us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
0507 02:07:04.190035 (+ 82us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.31.131.126:39473: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":68,"threads_started":1}
I20240507 02:07:19.196350 5049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:19.194218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55936 (local address 127.31.131.126:39473)
0507 02:07:19.194338 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:19.194341 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:19.194373 (+ 32us) server_negotiation.cc:407] Connection header received
0507 02:07:19.194453 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:19.194454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:19.194466 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:19.194520 (+ 54us) server_negotiation.cc:226] Negotiated authn=JWT
0507 02:07:19.194792 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:19.195297 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:19.195847 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:19.195980 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:19.195999 (+ 19us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0507 02:07:19.196170 (+ 171us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
0507 02:07:19.196280 (+ 110us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55936: JWT verification failed: token expired
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":52,"threads_started":1}
W20240507 02:07:19.196391 5048 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:07:19.194137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:39473 (local address 127.0.0.1:55936)
0507 02:07:19.194292 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:19.194303 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:19.194382 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:19.194528 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:19.194529 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:19.194531 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0507 02:07:19.194723 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:19.194725 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:19.195371 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:19.195373 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:19.195771 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:19.195772 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:19.195832 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:19.195839 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0507 02:07:19.196238 (+ 399us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
0507 02:07:19.196326 (+ 88us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.31.131.126:39473: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":98,"threads_started":1}
W20240507 02:07:19.196435 5049 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55936: JWT verification failed: token expired
W20240507 02:07:19.197633 5049 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:07:19.197225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55946 (local address 127.31.131.126:39473)
0507 02:07:19.197271 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:19.197273 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:19.197340 (+ 67us) server_negotiation.cc:407] Connection header received
0507 02:07:19.197404 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:19.197405 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:19.197415 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:19.197461 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:19.197585 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55946: BlockingRecv error: recv got EOF from 127.0.0.1:55946 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20240507 02:07:19.200001 5059 client_negotiation.cc:375] the client has a JWT but it isn't advertising its JWT-based authn capability since it doesn't trust the certificate of the server at 127.31.131.126:39473
W20240507 02:07:19.200289 5049 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:07:19.199885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55952 (local address 127.31.131.126:39473)
0507 02:07:19.199930 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:19.199931 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:19.199976 (+ 45us) server_negotiation.cc:407] Connection header received
0507 02:07:19.200112 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:19.200113 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:19.200122 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:19.200177 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:19.200250 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55952: BlockingRecv error: recv got EOF from 127.0.0.1:55952 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
I20240507 02:07:19.200371 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 4939
2024-05-07T02:07:19Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniCluster (15253 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20240507 02:07:19.237282 32269 mini_oidc.cc:152] Starting JWKS server
I20240507 02:07:19.237965 32269 webserver.cc:460] Webserver started at https://127.0.0.1:33925/ using document root <none> and password file <none>
I20240507 02:07:19.238010 32269 mini_oidc.cc:189] Starting OIDC Discovery server
I20240507 02:07:19.238134 32269 webserver.cc:460] Webserver started at http://127.0.0.1:37065/ using document root <none> and password file <none>
2024-05-07T02:07:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-07T02:07:19Z Disabled control of system clock
I20240507 02:07:19.243786 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1715047604443657-32269-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1715047604443657-32269-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1715047604443657-32269-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:39173
--webserver_interface=127.31.131.126
--webserver_port=0
--builtin_ntp_servers=127.31.131.84:43947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.131.126:39173
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33925/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1715047604443657-32269-0/testchainca.cert with env {}
W20240507 02:07:19.318130 5065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:07:19.318267 5065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:07:19.318293 5065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:07:19.319525 5065 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240507 02:07:19.319563 5065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:07:19.319579 5065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:07:19.319595 5065 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240507 02:07:19.319609 5065 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20240507 02:07:19.319631 5065 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20240507 02:07:19.319648 5065 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33925/jwks.json
I20240507 02:07:19.321185 5065 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:43947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1715047604443657-32269-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1715047604443657-32269-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.131.126:39173
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:39173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33925/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.131.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1715047604443657-32269-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1715047604443657-32269-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:07:19.321370 5065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:07:19.321513 5065 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:07:19.325047 5073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:07:19.325084 5071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:07:19.325191 5065 server_base.cc:1034] running on GCE node
W20240507 02:07:19.325248 5070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:07:19.325438 5065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:07:19.325634 5065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:07:19.326753 5065 hybrid_clock.cc:648] HybridClock initialized: now 1715047639326741 us; error 28 us; skew 500 ppm
I20240507 02:07:19.327782 5065 webserver.cc:460] Webserver started at http://127.31.131.126:42387/ using document root <none> and password file <none>
I20240507 02:07:19.327956 5065 fs_manager.cc:359] Metadata directory not provided
I20240507 02:07:19.327993 5065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:07:19.328083 5065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:07:19.328917 5065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1715047604443657-32269-0/minicluster-data/master-0/data/instance:
uuid: "cb6d3789887b4975826fbfcfe16a5fb9"
format_stamp: "Formatted at 2024-05-07 02:07:19 on dist-test-slave-v0bg"
server_key: "606db4ee8b729c0383cf57d6c6fee486"
server_key_iv: "f8dae04281dc3f4e84ef214479f39d13"
server_key_version: "encryptionkey@0"
I20240507 02:07:19.329183 5065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1715047604443657-32269-0/minicluster-data/master-0/wal/instance:
uuid: "cb6d3789887b4975826fbfcfe16a5fb9"
format_stamp: "Formatted at 2024-05-07 02:07:19 on dist-test-slave-v0bg"
server_key: "606db4ee8b729c0383cf57d6c6fee486"
server_key_iv: "f8dae04281dc3f4e84ef214479f39d13"
server_key_version: "encryptionkey@0"
I20240507 02:07:19.330150 5065 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240507 02:07:19.330693 5079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:19.330832 5065 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:19.330909 5065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1715047604443657-32269-0/minicluster-data/master-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1715047604443657-32269-0/minicluster-data/master-0/wal
uuid: "cb6d3789887b4975826fbfcfe16a5fb9"
format_stamp: "Formatted at 2024-05-07 02:07:19 on dist-test-slave-v0bg"
server_key: "606db4ee8b729c0383cf57d6c6fee486"
server_key_iv: "f8dae04281dc3f4e84ef214479f39d13"
server_key_version: "encryptionkey@0"
I20240507 02:07:19.330962 5065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1715047604443657-32269-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1715047604443657-32269-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1715047604443657-32269-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:07:19.352948 5065 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:07:19.353127 5065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:07:19.353215 5065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:07:19.356107 5065 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.126:39173
I20240507 02:07:19.356133 5131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.126:39173 every 8 connection(s)
I20240507 02:07:19.356374 5065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
I20240507 02:07:19.356598 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 5065
I20240507 02:07:19.356685 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1715047604443657-32269-0/minicluster-data/master-0/wal/instance
I20240507 02:07:19.356833 32269 external_mini_cluster.cc:1428] Setting key 4a479ec4a158b629a9e57dfcecd4ceac
I20240507 02:07:19.356925 5132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:07:19.359194 5132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb6d3789887b4975826fbfcfe16a5fb9: Bootstrap starting.
I20240507 02:07:19.359630 5132 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb6d3789887b4975826fbfcfe16a5fb9: Neither blocks nor log segments found. Creating new log.
I20240507 02:07:19.359872 5132 log.cc:824] T 00000000000000000000000000000000 P cb6d3789887b4975826fbfcfe16a5fb9: Log is configured to *not* fsync() on all Append() calls
I20240507 02:07:19.360450 5132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb6d3789887b4975826fbfcfe16a5fb9: No bootstrap required, opened a new log
I20240507 02:07:19.360658 5135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:19.357218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46284 (local address 127.31.131.126:39173)
0507 02:07:19.357421 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:19.357427 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:19.357444 (+ 17us) server_negotiation.cc:407] Connection header received
0507 02:07:19.357792 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:19.357796 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:19.357839 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:19.357953 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:19.358289 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:19.358993 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:19.359542 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:19.359762 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:19.360120 (+ 358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:19.360128 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:19.360128 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0507 02:07:19.360281 (+ 153us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:19.360304 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0507 02:07:19.360309 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:19.360367 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:19.360448 (+ 81us) server_negotiation.cc:299] Negotiation successful
0507 02:07:19.360492 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":80,"threads_started":1}
I20240507 02:07:19.361498 32269 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240507 02:07:19.362296 5132 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb6d3789887b4975826fbfcfe16a5fb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb6d3789887b4975826fbfcfe16a5fb9" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 39173 } }
I20240507 02:07:19.362428 5132 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb6d3789887b4975826fbfcfe16a5fb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:07:19.362448 5132 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb6d3789887b4975826fbfcfe16a5fb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb6d3789887b4975826fbfcfe16a5fb9, State: Initialized, Role: FOLLOWER
I20240507 02:07:19.362514 5132 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb6d3789887b4975826fbfcfe16a5fb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb6d3789887b4975826fbfcfe16a5fb9" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 39173 } }
I20240507 02:07:19.362593 5132 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb6d3789887b4975826fbfcfe16a5fb9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 02:07:19.362623 5132 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb6d3789887b4975826fbfcfe16a5fb9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 02:07:19.362651 5132 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb6d3789887b4975826fbfcfe16a5fb9 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:07:19.363070 5132 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb6d3789887b4975826fbfcfe16a5fb9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb6d3789887b4975826fbfcfe16a5fb9" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 39173 } }
I20240507 02:07:19.363145 5132 leader_election.cc:302] T 00000000000000000000000000000000 P cb6d3789887b4975826fbfcfe16a5fb9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb6d3789887b4975826fbfcfe16a5fb9; no voters:
I20240507 02:07:19.363245 5132 leader_election.cc:288] T 00000000000000000000000000000000 P cb6d3789887b4975826fbfcfe16a5fb9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240507 02:07:19.363301 5137 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb6d3789887b4975826fbfcfe16a5fb9 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:07:19.363408 5132 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb6d3789887b4975826fbfcfe16a5fb9 [sys.catalog]: configured and running, proceeding with master startup.
I20240507 02:07:19.363443 5137 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb6d3789887b4975826fbfcfe16a5fb9 [term 1 LEADER]: Becoming Leader. State: Replica: cb6d3789887b4975826fbfcfe16a5fb9, State: Running, Role: LEADER
I20240507 02:07:19.363623 5137 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb6d3789887b4975826fbfcfe16a5fb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb6d3789887b4975826fbfcfe16a5fb9" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 39173 } }
I20240507 02:07:19.363938 5139 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb6d3789887b4975826fbfcfe16a5fb9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb6d3789887b4975826fbfcfe16a5fb9. Latest consensus state: current_term: 1 leader_uuid: "cb6d3789887b4975826fbfcfe16a5fb9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb6d3789887b4975826fbfcfe16a5fb9" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 39173 } } }
I20240507 02:07:19.364014 5139 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb6d3789887b4975826fbfcfe16a5fb9 [sys.catalog]: This master's current role is: LEADER
I20240507 02:07:19.364146 5138 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb6d3789887b4975826fbfcfe16a5fb9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb6d3789887b4975826fbfcfe16a5fb9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb6d3789887b4975826fbfcfe16a5fb9" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 39173 } } }
I20240507 02:07:19.364182 5146 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 02:07:19.364220 5138 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb6d3789887b4975826fbfcfe16a5fb9 [sys.catalog]: This master's current role is: LEADER
I20240507 02:07:19.364588 5146 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240507 02:07:19.365842 5146 catalog_manager.cc:1347] Generated new cluster ID: d54ebecc30384227974a68dff2e1fae3
I20240507 02:07:19.365892 5146 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240507 02:07:19.382455 5146 catalog_manager.cc:1370] Generated new certificate authority record
I20240507 02:07:19.382908 5146 catalog_manager.cc:1504] Loading token signing keys...
I20240507 02:07:19.385597 5146 catalog_manager.cc:5857] T 00000000000000000000000000000000 P cb6d3789887b4975826fbfcfe16a5fb9: Generated new TSK 0
I20240507 02:07:19.385730 5146 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20240507 02:07:19.410741 5162 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20240507 02:07:19.410806 5162 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20240507 02:07:19.411038 5135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:19.396646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46286 (local address 127.31.131.126:39173)
0507 02:07:19.396690 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:19.396692 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:19.396758 (+ 66us) server_negotiation.cc:407] Connection header received
0507 02:07:19.396825 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:19.396826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:19.396846 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:19.396899 (+ 53us) server_negotiation.cc:226] Negotiated authn=JWT
0507 02:07:19.397197 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:19.397651 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:19.398450 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:19.398600 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:19.398636 (+ 36us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0507 02:07:19.410869 (+ 12233us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33925/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0507 02:07:19.410990 (+ 121us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46286: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33925/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":26}
W20240507 02:07:19.411077 5135 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46286: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33925/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20240507 02:07:19.411131 5161 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:07:19.396555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:39173 (local address 127.0.0.1:46286)
0507 02:07:19.396680 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:19.396690 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:19.396760 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:19.396910 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:19.396911 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:19.396913 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0507 02:07:19.397136 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:19.397138 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:19.397725 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:19.397725 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:19.398354 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:19.398356 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:19.398448 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:19.398452 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0507 02:07:19.410953 (+ 12501us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33925/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0507 02:07:19.411047 (+ 94us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.31.131.126:39173: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33925/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":70,"threads_started":1}
I20240507 02:07:19.411355 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 5065
2024-05-07T02:07:19Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (210 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20240507 02:07:19.447667 32269 mini_oidc.cc:152] Starting JWKS server
I20240507 02:07:19.448280 32269 webserver.cc:460] Webserver started at https://127.0.0.1:43607/ using document root <none> and password file <none>
I20240507 02:07:19.448316 32269 mini_oidc.cc:189] Starting OIDC Discovery server
I20240507 02:07:19.448421 32269 webserver.cc:460] Webserver started at http://127.0.0.1:44581/ using document root <none> and password file <none>
2024-05-07T02:07:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-07T02:07:19Z Disabled control of system clock
I20240507 02:07:19.453991 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1715047604443657-32269-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1715047604443657-32269-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1715047604443657-32269-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:39879
--webserver_interface=127.31.131.126
--webserver_port=0
--builtin_ntp_servers=127.31.131.84:45931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.131.126:39879
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43607/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20240507 02:07:19.526029 5168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:07:19.526177 5168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:07:19.526204 5168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:07:19.527446 5168 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240507 02:07:19.527483 5168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:07:19.527499 5168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:07:19.527514 5168 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240507 02:07:19.527529 5168 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20240507 02:07:19.527545 5168 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20240507 02:07:19.527568 5168 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43607/jwks.json
I20240507 02:07:19.528928 5168 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:45931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1715047604443657-32269-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1715047604443657-32269-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.131.126:39879
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:39879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43607/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.131.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5168
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1715047604443657-32269-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:07:19.529099 5168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:07:19.529284 5168 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:07:19.532728 5174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:07:19.532729 5176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:07:19.532900 5173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:07:19.533085 5168 server_base.cc:1034] running on GCE node
I20240507 02:07:19.533272 5168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:07:19.533495 5168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:07:19.534606 5168 hybrid_clock.cc:648] HybridClock initialized: now 1715047639534593 us; error 29 us; skew 500 ppm
I20240507 02:07:19.535851 5168 webserver.cc:460] Webserver started at http://127.31.131.126:37077/ using document root <none> and password file <none>
I20240507 02:07:19.536031 5168 fs_manager.cc:359] Metadata directory not provided
I20240507 02:07:19.536082 5168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:07:19.536185 5168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:07:19.537318 5168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1715047604443657-32269-0/minicluster-data/master-0/data/instance:
uuid: "99d53cb03c0f4337bededf08a1633276"
format_stamp: "Formatted at 2024-05-07 02:07:19 on dist-test-slave-v0bg"
server_key: "6be31b778d9f339467e3878fa6a9866b"
server_key_iv: "5f29e462de3719b7239b74bcc7dff9a2"
server_key_version: "encryptionkey@0"
I20240507 02:07:19.537622 5168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1715047604443657-32269-0/minicluster-data/master-0/wal/instance:
uuid: "99d53cb03c0f4337bededf08a1633276"
format_stamp: "Formatted at 2024-05-07 02:07:19 on dist-test-slave-v0bg"
server_key: "6be31b778d9f339467e3878fa6a9866b"
server_key_iv: "5f29e462de3719b7239b74bcc7dff9a2"
server_key_version: "encryptionkey@0"
I20240507 02:07:19.538786 5168 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240507 02:07:19.539456 5182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:19.539613 5168 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240507 02:07:19.539685 5168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1715047604443657-32269-0/minicluster-data/master-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1715047604443657-32269-0/minicluster-data/master-0/wal
uuid: "99d53cb03c0f4337bededf08a1633276"
format_stamp: "Formatted at 2024-05-07 02:07:19 on dist-test-slave-v0bg"
server_key: "6be31b778d9f339467e3878fa6a9866b"
server_key_iv: "5f29e462de3719b7239b74bcc7dff9a2"
server_key_version: "encryptionkey@0"
I20240507 02:07:19.539741 5168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1715047604443657-32269-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1715047604443657-32269-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1715047604443657-32269-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:07:19.546687 5168 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:07:19.546866 5168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:07:19.546943 5168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:07:19.550148 5168 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.126:39879
I20240507 02:07:19.550175 5235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.126:39879 every 8 connection(s)
I20240507 02:07:19.550401 5168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
I20240507 02:07:19.550881 5236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:07:19.552860 5236 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99d53cb03c0f4337bededf08a1633276: Bootstrap starting.
I20240507 02:07:19.553299 5236 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99d53cb03c0f4337bededf08a1633276: Neither blocks nor log segments found. Creating new log.
I20240507 02:07:19.553519 5236 log.cc:824] T 00000000000000000000000000000000 P 99d53cb03c0f4337bededf08a1633276: Log is configured to *not* fsync() on all Append() calls
I20240507 02:07:19.554167 5236 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99d53cb03c0f4337bededf08a1633276: No bootstrap required, opened a new log
I20240507 02:07:19.555562 5236 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99d53cb03c0f4337bededf08a1633276 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99d53cb03c0f4337bededf08a1633276" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 39879 } }
I20240507 02:07:19.555670 5236 raft_consensus.cc:374] T 00000000000000000000000000000000 P 99d53cb03c0f4337bededf08a1633276 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:07:19.555691 5236 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99d53cb03c0f4337bededf08a1633276 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99d53cb03c0f4337bededf08a1633276, State: Initialized, Role: FOLLOWER
I20240507 02:07:19.555800 5236 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99d53cb03c0f4337bededf08a1633276 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99d53cb03c0f4337bededf08a1633276" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 39879 } }
I20240507 02:07:19.555866 5236 raft_consensus.cc:388] T 00000000000000000000000000000000 P 99d53cb03c0f4337bededf08a1633276 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 02:07:19.555899 5236 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99d53cb03c0f4337bededf08a1633276 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 02:07:19.555943 5236 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99d53cb03c0f4337bededf08a1633276 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:07:19.556452 5236 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99d53cb03c0f4337bededf08a1633276 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99d53cb03c0f4337bededf08a1633276" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 39879 } }
I20240507 02:07:19.556540 5236 leader_election.cc:302] T 00000000000000000000000000000000 P 99d53cb03c0f4337bededf08a1633276 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99d53cb03c0f4337bededf08a1633276; no voters:
I20240507 02:07:19.556679 5236 leader_election.cc:288] T 00000000000000000000000000000000 P 99d53cb03c0f4337bededf08a1633276 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240507 02:07:19.556716 5239 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99d53cb03c0f4337bededf08a1633276 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:07:19.556818 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 5168
I20240507 02:07:19.556835 5239 raft_consensus.cc:686] T 00000000000000000000000000000000 P 99d53cb03c0f4337bededf08a1633276 [term 1 LEADER]: Becoming Leader. State: Replica: 99d53cb03c0f4337bededf08a1633276, State: Running, Role: LEADER
I20240507 02:07:19.556910 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1715047604443657-32269-0/minicluster-data/master-0/wal/instance
I20240507 02:07:19.556923 5236 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99d53cb03c0f4337bededf08a1633276 [sys.catalog]: configured and running, proceeding with master startup.
I20240507 02:07:19.556913 5239 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99d53cb03c0f4337bededf08a1633276 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99d53cb03c0f4337bededf08a1633276" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 39879 } }
I20240507 02:07:19.557083 32269 external_mini_cluster.cc:1428] Setting key 41c9315da7b519be4dc9ada58c83ac41
I20240507 02:07:19.557215 5239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99d53cb03c0f4337bededf08a1633276 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99d53cb03c0f4337bededf08a1633276. Latest consensus state: current_term: 1 leader_uuid: "99d53cb03c0f4337bededf08a1633276" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99d53cb03c0f4337bededf08a1633276" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 39879 } } }
I20240507 02:07:19.557303 5239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99d53cb03c0f4337bededf08a1633276 [sys.catalog]: This master's current role is: LEADER
I20240507 02:07:19.557230 5240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99d53cb03c0f4337bededf08a1633276 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "99d53cb03c0f4337bededf08a1633276" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99d53cb03c0f4337bededf08a1633276" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 39879 } } }
I20240507 02:07:19.557411 5240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99d53cb03c0f4337bededf08a1633276 [sys.catalog]: This master's current role is: LEADER
I20240507 02:07:19.557600 5244 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 02:07:19.557986 5244 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240507 02:07:19.559327 5244 catalog_manager.cc:1347] Generated new cluster ID: 975298e23e8444188b8f57134f998aa5
I20240507 02:07:19.559378 5244 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240507 02:07:19.562269 5250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:19.557608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54860 (local address 127.31.131.126:39879)
0507 02:07:19.559032 (+ 1424us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:19.559039 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:19.559060 (+ 21us) server_negotiation.cc:407] Connection header received
0507 02:07:19.559147 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:19.559152 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:19.559203 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:19.559370 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:19.559738 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:19.560449 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:19.560988 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:19.561547 (+ 559us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:19.561632 (+ 85us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:19.561649 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:19.561650 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0507 02:07:19.561795 (+ 145us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:19.561813 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0507 02:07:19.561819 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:19.562009 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:19.562078 (+ 69us) server_negotiation.cc:299] Negotiation successful
0507 02:07:19.562124 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1360,"thread_start_us":67,"threads_started":1}
I20240507 02:07:19.562995 32269 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240507 02:07:19.570434 5244 catalog_manager.cc:1370] Generated new certificate authority record
I20240507 02:07:19.570914 5244 catalog_manager.cc:1504] Loading token signing keys...
I20240507 02:07:19.575954 5244 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 99d53cb03c0f4337bededf08a1633276: Generated new TSK 0
I20240507 02:07:19.576081 5244 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20240507 02:07:19.602309 5266 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20240507 02:07:19.602365 5266 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20240507 02:07:19.603098 5250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:19.583988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54862 (local address 127.31.131.126:39879)
0507 02:07:19.584030 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:19.584031 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:19.584114 (+ 83us) server_negotiation.cc:407] Connection header received
0507 02:07:19.584176 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:19.584177 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:19.584188 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:19.584242 (+ 54us) server_negotiation.cc:226] Negotiated authn=JWT
0507 02:07:19.584460 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:19.584924 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:19.585448 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:19.585588 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:19.585620 (+ 32us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0507 02:07:19.602904 (+ 17284us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43607/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0507 02:07:19.603002 (+ 98us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54862: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43607/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":22}
W20240507 02:07:19.603122 5265 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:07:19.583929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:39879 (local address 127.0.0.1:54862)
0507 02:07:19.584048 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:19.584057 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:19.584115 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:19.584246 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:19.584247 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:19.584249 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0507 02:07:19.584405 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:19.584406 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:19.584998 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:19.584999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:19.585385 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:19.585387 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:19.585445 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:19.585450 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0507 02:07:19.602980 (+ 17530us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43607/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0507 02:07:19.603056 (+ 76us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.31.131.126:39879: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43607/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":67,"threads_started":1}
W20240507 02:07:19.603150 5250 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54862: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43607/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20240507 02:07:19.603440 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 5168
2024-05-07T02:07:19Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (191 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2024-05-07T02:07:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-07T02:07:19Z Disabled control of system clock
I20240507 02:07:19.644863 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1715047604443657-32269-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1715047604443657-32269-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1715047604443657-32269-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:45799
--webserver_interface=127.31.131.126
--webserver_port=0
--builtin_ntp_servers=127.31.131.84:33539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.131.126:45799
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1715047604443657-32269-0/testchainca.cert with env {}
W20240507 02:07:19.716959 5270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:07:19.717103 5270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:07:19.717130 5270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:07:19.718322 5270 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240507 02:07:19.718359 5270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:07:19.718370 5270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:07:19.718381 5270 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240507 02:07:19.718401 5270 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20240507 02:07:19.718412 5270 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20240507 02:07:19.718421 5270 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20240507 02:07:19.719939 5270 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:33539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1715047604443657-32269-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1715047604443657-32269-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.131.126:45799
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:45799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.131.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1715047604443657-32269-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1715047604443657-32269-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:07:19.720085 5270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:07:19.720207 5270 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:07:19.723528 5276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:07:19.723567 5278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:07:19.723615 5270 server_base.cc:1034] running on GCE node
W20240507 02:07:19.723708 5275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:07:19.723927 5270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:07:19.724117 5270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:07:19.725240 5270 hybrid_clock.cc:648] HybridClock initialized: now 1715047639725227 us; error 32 us; skew 500 ppm
I20240507 02:07:19.726258 5270 webserver.cc:460] Webserver started at http://127.31.131.126:41781/ using document root <none> and password file <none>
I20240507 02:07:19.726413 5270 fs_manager.cc:359] Metadata directory not provided
I20240507 02:07:19.726450 5270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:07:19.726540 5270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:07:19.727396 5270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1715047604443657-32269-0/minicluster-data/master-0/data/instance:
uuid: "638731b316ad41c6b8457fa5e1d80be6"
format_stamp: "Formatted at 2024-05-07 02:07:19 on dist-test-slave-v0bg"
server_key: "a1dc87540c67b158e83edacb6c3d0184"
server_key_iv: "de05e2c12512c8398e1a6ff786166244"
server_key_version: "encryptionkey@0"
I20240507 02:07:19.727648 5270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1715047604443657-32269-0/minicluster-data/master-0/wal/instance:
uuid: "638731b316ad41c6b8457fa5e1d80be6"
format_stamp: "Formatted at 2024-05-07 02:07:19 on dist-test-slave-v0bg"
server_key: "a1dc87540c67b158e83edacb6c3d0184"
server_key_iv: "de05e2c12512c8398e1a6ff786166244"
server_key_version: "encryptionkey@0"
I20240507 02:07:19.728616 5270 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240507 02:07:19.729159 5284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:19.729302 5270 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240507 02:07:19.729369 5270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1715047604443657-32269-0/minicluster-data/master-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1715047604443657-32269-0/minicluster-data/master-0/wal
uuid: "638731b316ad41c6b8457fa5e1d80be6"
format_stamp: "Formatted at 2024-05-07 02:07:19 on dist-test-slave-v0bg"
server_key: "a1dc87540c67b158e83edacb6c3d0184"
server_key_iv: "de05e2c12512c8398e1a6ff786166244"
server_key_version: "encryptionkey@0"
I20240507 02:07:19.729420 5270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1715047604443657-32269-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1715047604443657-32269-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1715047604443657-32269-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:07:19.752661 5270 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:07:19.752854 5270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:07:19.752954 5270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:07:19.755899 5270 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.126:45799
I20240507 02:07:19.755928 5337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.126:45799 every 8 connection(s)
I20240507 02:07:19.756170 5270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
I20240507 02:07:19.756654 5338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:07:19.757666 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 5270
I20240507 02:07:19.757751 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1715047604443657-32269-0/minicluster-data/master-0/wal/instance
I20240507 02:07:19.757896 32269 external_mini_cluster.cc:1428] Setting key 8bf6ad7e264d9b72c214f0e146172bae
I20240507 02:07:19.759114 5338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 638731b316ad41c6b8457fa5e1d80be6: Bootstrap starting.
I20240507 02:07:19.759685 5338 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 638731b316ad41c6b8457fa5e1d80be6: Neither blocks nor log segments found. Creating new log.
I20240507 02:07:19.759919 5338 log.cc:824] T 00000000000000000000000000000000 P 638731b316ad41c6b8457fa5e1d80be6: Log is configured to *not* fsync() on all Append() calls
I20240507 02:07:19.760550 5338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 638731b316ad41c6b8457fa5e1d80be6: No bootstrap required, opened a new log
I20240507 02:07:19.761869 5338 raft_consensus.cc:348] T 00000000000000000000000000000000 P 638731b316ad41c6b8457fa5e1d80be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "638731b316ad41c6b8457fa5e1d80be6" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 45799 } }
I20240507 02:07:19.762009 5338 raft_consensus.cc:374] T 00000000000000000000000000000000 P 638731b316ad41c6b8457fa5e1d80be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:07:19.762032 5338 raft_consensus.cc:729] T 00000000000000000000000000000000 P 638731b316ad41c6b8457fa5e1d80be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 638731b316ad41c6b8457fa5e1d80be6, State: Initialized, Role: FOLLOWER
I20240507 02:07:19.762116 5338 consensus_queue.cc:260] T 00000000000000000000000000000000 P 638731b316ad41c6b8457fa5e1d80be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "638731b316ad41c6b8457fa5e1d80be6" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 45799 } }
I20240507 02:07:19.762120 5341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:19.758289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49038 (local address 127.31.131.126:45799)
0507 02:07:19.758475 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:19.758482 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:19.758501 (+ 19us) server_negotiation.cc:407] Connection header received
0507 02:07:19.759030 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:19.759037 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:19.759098 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:19.759239 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:19.759527 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:19.760484 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:19.761030 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:19.761250 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:19.761524 (+ 274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:19.761533 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:19.761533 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0507 02:07:19.761663 (+ 130us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:19.761690 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0507 02:07:19.761696 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:19.761754 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:19.761830 (+ 76us) server_negotiation.cc:299] Negotiation successful
0507 02:07:19.761882 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":69,"threads_started":1}
I20240507 02:07:19.762180 5338 raft_consensus.cc:388] T 00000000000000000000000000000000 P 638731b316ad41c6b8457fa5e1d80be6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 02:07:19.762207 5338 raft_consensus.cc:482] T 00000000000000000000000000000000 P 638731b316ad41c6b8457fa5e1d80be6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 02:07:19.762240 5338 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 638731b316ad41c6b8457fa5e1d80be6 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:07:19.762739 5338 raft_consensus.cc:504] T 00000000000000000000000000000000 P 638731b316ad41c6b8457fa5e1d80be6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "638731b316ad41c6b8457fa5e1d80be6" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 45799 } }
I20240507 02:07:19.762833 5338 leader_election.cc:302] T 00000000000000000000000000000000 P 638731b316ad41c6b8457fa5e1d80be6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 638731b316ad41c6b8457fa5e1d80be6; no voters:
I20240507 02:07:19.762970 5338 leader_election.cc:288] T 00000000000000000000000000000000 P 638731b316ad41c6b8457fa5e1d80be6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240507 02:07:19.762964 32269 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240507 02:07:19.763005 5343 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 638731b316ad41c6b8457fa5e1d80be6 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:07:19.763108 5343 raft_consensus.cc:686] T 00000000000000000000000000000000 P 638731b316ad41c6b8457fa5e1d80be6 [term 1 LEADER]: Becoming Leader. State: Replica: 638731b316ad41c6b8457fa5e1d80be6, State: Running, Role: LEADER
I20240507 02:07:19.763172 5338 sys_catalog.cc:564] T 00000000000000000000000000000000 P 638731b316ad41c6b8457fa5e1d80be6 [sys.catalog]: configured and running, proceeding with master startup.
I20240507 02:07:19.763187 5343 consensus_queue.cc:237] T 00000000000000000000000000000000 P 638731b316ad41c6b8457fa5e1d80be6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "638731b316ad41c6b8457fa5e1d80be6" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 45799 } }
I20240507 02:07:19.763481 5343 sys_catalog.cc:455] T 00000000000000000000000000000000 P 638731b316ad41c6b8457fa5e1d80be6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 638731b316ad41c6b8457fa5e1d80be6. Latest consensus state: current_term: 1 leader_uuid: "638731b316ad41c6b8457fa5e1d80be6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "638731b316ad41c6b8457fa5e1d80be6" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 45799 } } }
I20240507 02:07:19.763553 5343 sys_catalog.cc:458] T 00000000000000000000000000000000 P 638731b316ad41c6b8457fa5e1d80be6 [sys.catalog]: This master's current role is: LEADER
I20240507 02:07:19.763631 5345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 638731b316ad41c6b8457fa5e1d80be6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "638731b316ad41c6b8457fa5e1d80be6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "638731b316ad41c6b8457fa5e1d80be6" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 45799 } } }
I20240507 02:07:19.763718 5345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 638731b316ad41c6b8457fa5e1d80be6 [sys.catalog]: This master's current role is: LEADER
I20240507 02:07:19.763864 5348 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 02:07:19.764302 5348 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240507 02:07:19.765444 5348 catalog_manager.cc:1347] Generated new cluster ID: b1bed0a177d94735915ce88eb511525e
I20240507 02:07:19.765486 5348 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240507 02:07:19.772663 5348 catalog_manager.cc:1370] Generated new certificate authority record
I20240507 02:07:19.773360 5348 catalog_manager.cc:1504] Loading token signing keys...
I20240507 02:07:19.780481 5348 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 638731b316ad41c6b8457fa5e1d80be6: Generated new TSK 0
I20240507 02:07:19.780620 5348 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240507 02:07:19.805989 5341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:19.793793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49044 (local address 127.31.131.126:45799)
0507 02:07:19.793827 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:19.793829 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:19.793919 (+ 90us) server_negotiation.cc:407] Connection header received
0507 02:07:19.793982 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:19.793982 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:19.793990 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:19.794042 (+ 52us) server_negotiation.cc:226] Negotiated authn=JWT
0507 02:07:19.794337 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:19.794822 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:19.795603 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:19.795716 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:19.795747 (+ 31us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0507 02:07:19.805840 (+ 10093us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0507 02:07:19.805949 (+ 109us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49044: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":19}
W20240507 02:07:19.806030 5341 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49044: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20240507 02:07:19.806115 5367 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:07:19.793712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:45799 (local address 127.0.0.1:49044)
0507 02:07:19.793846 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:19.793861 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:19.793926 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:19.794052 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:19.794052 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:19.794054 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0507 02:07:19.794283 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:19.794285 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:19.794899 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:19.794900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:19.795533 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:19.795535 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:19.795605 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:19.795610 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0507 02:07:19.805929 (+ 10319us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0507 02:07:19.806028 (+ 99us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.31.131.126:45799: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":77,"threads_started":1}
I20240507 02:07:19.806351 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 5270
2024-05-07T02:07:19Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (182 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (78 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (77 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 07 02:07:20 dist-test-slave-v0bg krb5kdc[5379](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 07 02:07:20 dist-test-slave-v0bg krb5kdc[5379](info): set up 2 sockets
May 07 02:07:20 dist-test-slave-v0bg krb5kdc[5379](info): commencing operation
krb5kdc: starting...
W20240507 02:07:20.839280 32269 mini_kdc.cc:121] Time spent starting KDC: real 0.861s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/krb5kdc/test-user.keytab.
May 07 02:07:20 dist-test-slave-v0bg krb5kdc[5379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-07T02:07:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-07T02:07:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.131.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab.
Entry for principal kudu/127.31.131.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab.
I20240507 02:07:20.934729 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:35425
--webserver_interface=127.31.131.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab
--principal=kudu/127.31.131.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:45965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.131.126:35425
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:07:21.007635 5392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:07:21.007783 5392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:07:21.007813 5392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:07:21.009019 5392 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240507 02:07:21.009056 5392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:07:21.009074 5392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:07:21.009088 5392 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240507 02:07:21.009102 5392 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240507 02:07:21.010636 5392 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:45965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.131.126:35425
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab
--principal=kudu/127.31.131.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:35425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.131.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5392
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:07:21.010824 5392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:07:21.010982 5392 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:07:21.014355 5397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:07:21.014376 5400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:07:21.014461 5398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:07:21.014484 5392 server_base.cc:1034] running on GCE node
I20240507 02:07:21.014755 5392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:07:21.014974 5392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:07:21.016098 5392 hybrid_clock.cc:648] HybridClock initialized: now 1715047641016084 us; error 29 us; skew 500 ppm
May 07 02:07:21 dist-test-slave-v0bg krb5kdc[5379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047641, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:07:21.018124 5392 init.cc:377] Logged in from keytab as kudu/127.31.131.126@KRBTEST.COM (short username kudu)
I20240507 02:07:21.018488 5392 webserver.cc:460] Webserver started at http://127.31.131.126:38473/ using document root <none> and password file <none>
I20240507 02:07:21.018666 5392 fs_manager.cc:359] Metadata directory not provided
I20240507 02:07:21.018707 5392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:07:21.018800 5392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:07:21.019675 5392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/minicluster-data/master-0/data/instance:
uuid: "a9757323d3d94474964513139550b40c"
format_stamp: "Formatted at 2024-05-07 02:07:21 on dist-test-slave-v0bg"
server_key: "bbd1a2d000557acde7597f14430da490"
server_key_iv: "e02de2043085232405de1841eefa7a1d"
server_key_version: "encryptionkey@0"
I20240507 02:07:21.019958 5392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/minicluster-data/master-0/wal/instance:
uuid: "a9757323d3d94474964513139550b40c"
format_stamp: "Formatted at 2024-05-07 02:07:21 on dist-test-slave-v0bg"
server_key: "bbd1a2d000557acde7597f14430da490"
server_key_iv: "e02de2043085232405de1841eefa7a1d"
server_key_version: "encryptionkey@0"
I20240507 02:07:21.020952 5392 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240507 02:07:21.021492 5407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:21.021636 5392 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240507 02:07:21.021704 5392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/minicluster-data/master-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/minicluster-data/master-0/wal
uuid: "a9757323d3d94474964513139550b40c"
format_stamp: "Formatted at 2024-05-07 02:07:21 on dist-test-slave-v0bg"
server_key: "bbd1a2d000557acde7597f14430da490"
server_key_iv: "e02de2043085232405de1841eefa7a1d"
server_key_version: "encryptionkey@0"
I20240507 02:07:21.021759 5392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:07:21.035571 5392 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:07:21.036962 5392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:07:21.037060 5392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:07:21.040412 5392 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.126:35425
I20240507 02:07:21.040438 5460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.126:35425 every 8 connection(s)
I20240507 02:07:21.040686 5392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
I20240507 02:07:21.041160 5461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:07:21.043378 5461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9757323d3d94474964513139550b40c: Bootstrap starting.
I20240507 02:07:21.043812 5461 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9757323d3d94474964513139550b40c: Neither blocks nor log segments found. Creating new log.
I20240507 02:07:21.044004 5461 log.cc:824] T 00000000000000000000000000000000 P a9757323d3d94474964513139550b40c: Log is configured to *not* fsync() on all Append() calls
I20240507 02:07:21.044530 5461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9757323d3d94474964513139550b40c: No bootstrap required, opened a new log
I20240507 02:07:21.045807 5461 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9757323d3d94474964513139550b40c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9757323d3d94474964513139550b40c" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 35425 } }
I20240507 02:07:21.045912 5461 raft_consensus.cc:374] T 00000000000000000000000000000000 P a9757323d3d94474964513139550b40c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:07:21.045926 5461 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9757323d3d94474964513139550b40c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9757323d3d94474964513139550b40c, State: Initialized, Role: FOLLOWER
I20240507 02:07:21.046013 5461 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9757323d3d94474964513139550b40c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9757323d3d94474964513139550b40c" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 35425 } }
I20240507 02:07:21.046056 5461 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9757323d3d94474964513139550b40c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 02:07:21.046074 5461 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9757323d3d94474964513139550b40c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 02:07:21.046100 5461 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9757323d3d94474964513139550b40c [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:07:21.046532 5461 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9757323d3d94474964513139550b40c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9757323d3d94474964513139550b40c" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 35425 } }
I20240507 02:07:21.046638 5461 leader_election.cc:302] T 00000000000000000000000000000000 P a9757323d3d94474964513139550b40c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9757323d3d94474964513139550b40c; no voters:
I20240507 02:07:21.046743 5461 leader_election.cc:288] T 00000000000000000000000000000000 P a9757323d3d94474964513139550b40c [CANDIDATE]: Term 1 election: Requested vote from peers
I20240507 02:07:21.046797 5464 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9757323d3d94474964513139550b40c [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:07:21.046940 5461 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9757323d3d94474964513139550b40c [sys.catalog]: configured and running, proceeding with master startup.
I20240507 02:07:21.046948 5464 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9757323d3d94474964513139550b40c [term 1 LEADER]: Becoming Leader. State: Replica: a9757323d3d94474964513139550b40c, State: Running, Role: LEADER
I20240507 02:07:21.047037 5464 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9757323d3d94474964513139550b40c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9757323d3d94474964513139550b40c" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 35425 } }
I20240507 02:07:21.047367 5466 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9757323d3d94474964513139550b40c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9757323d3d94474964513139550b40c. Latest consensus state: current_term: 1 leader_uuid: "a9757323d3d94474964513139550b40c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9757323d3d94474964513139550b40c" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 35425 } } }
I20240507 02:07:21.047447 5466 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9757323d3d94474964513139550b40c [sys.catalog]: This master's current role is: LEADER
I20240507 02:07:21.047678 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 5392
I20240507 02:07:21.047633 5473 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 02:07:21.047791 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/minicluster-data/master-0/wal/instance
I20240507 02:07:21.047966 32269 external_mini_cluster.cc:1428] Setting key 91fb88fa2a7f50e7cd73553e69278eba
I20240507 02:07:21.048190 5473 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240507 02:07:21.048233 5465 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9757323d3d94474964513139550b40c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a9757323d3d94474964513139550b40c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9757323d3d94474964513139550b40c" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 35425 } } }
I20240507 02:07:21.048321 5465 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9757323d3d94474964513139550b40c [sys.catalog]: This master's current role is: LEADER
I20240507 02:07:21.049443 5473 catalog_manager.cc:1347] Generated new cluster ID: e8a5476472204177a202459d18984c83
I20240507 02:07:21.049494 5473 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
May 07 02:07:21 dist-test-slave-v0bg krb5kdc[5379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:07:21.055555 5483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:21.048487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38248 (local address 127.31.131.126:35425)
0507 02:07:21.048753 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:21.048766 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:21.048787 (+ 21us) server_negotiation.cc:407] Connection header received
0507 02:07:21.048884 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:21.048890 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:21.048938 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:21.049057 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:21.049822 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:21.050527 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:21.051045 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:21.051530 (+ 485us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:21.052983 (+ 1453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:21.052988 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:21.052988 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:07:21.053000 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:21.054398 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:21.054729 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:21.054731 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:21.054731 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:21.054771 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:21.054892 (+ 121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:21.054893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:21.054893 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:21.055064 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:21.055182 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:21.055282 (+ 100us) server_negotiation.cc:299] Negotiation successful
0507 02:07:21.055386 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":52,"threads_started":1}
I20240507 02:07:21.074906 5473 catalog_manager.cc:1370] Generated new certificate authority record
I20240507 02:07:21.075610 5473 catalog_manager.cc:1504] Loading token signing keys...
I20240507 02:07:21.081949 5473 catalog_manager.cc:5857] T 00000000000000000000000000000000 P a9757323d3d94474964513139550b40c: Generated new TSK 0
I20240507 02:07:21.082087 5473 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.131.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab.
Entry for principal kudu/127.31.131.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab.
I20240507 02:07:21.102591 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.65:0
--local_ip_for_outbound_sockets=127.31.131.65
--webserver_interface=127.31.131.65
--webserver_port=0
--tserver_master_addrs=127.31.131.126:35425
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab
--principal=kudu/127.31.131.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:45965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:07:21.175797 5487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:07:21.175956 5487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:07:21.175976 5487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:07:21.177197 5487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:07:21.177237 5487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:07:21.177258 5487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.65
I20240507 02:07:21.178848 5487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:45965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab
--principal=kudu/127.31.131.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.131.65
--webserver_port=0
--tserver_master_addrs=127.31.131.126:35425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.65
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:07:21.179025 5487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:07:21.179186 5487 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:07:21.183123 5495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:07:21.183130 5493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:07:21.183303 5492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:07:21.183543 5487 server_base.cc:1034] running on GCE node
I20240507 02:07:21.183697 5487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:07:21.183874 5487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:07:21.185003 5487 hybrid_clock.cc:648] HybridClock initialized: now 1715047641184989 us; error 29 us; skew 500 ppm
May 07 02:07:21 dist-test-slave-v0bg krb5kdc[5379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047641, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:07:21.187390 5487 init.cc:377] Logged in from keytab as kudu/127.31.131.65@KRBTEST.COM (short username kudu)
I20240507 02:07:21.187811 5487 webserver.cc:460] Webserver started at http://127.31.131.65:46635/ using document root <none> and password file <none>
I20240507 02:07:21.188005 5487 fs_manager.cc:359] Metadata directory not provided
I20240507 02:07:21.188051 5487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:07:21.188149 5487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:07:21.189337 5487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/minicluster-data/ts-0/data/instance:
uuid: "0cd50473bac94e81beb0660cbc43d2be"
format_stamp: "Formatted at 2024-05-07 02:07:21 on dist-test-slave-v0bg"
server_key: "1c9ea1dc2a9195bc1c54eb23e6778dc6"
server_key_iv: "94d4ece240903e0c4049ed6e279d439c"
server_key_version: "encryptionkey@0"
I20240507 02:07:21.189679 5487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/minicluster-data/ts-0/wal/instance:
uuid: "0cd50473bac94e81beb0660cbc43d2be"
format_stamp: "Formatted at 2024-05-07 02:07:21 on dist-test-slave-v0bg"
server_key: "1c9ea1dc2a9195bc1c54eb23e6778dc6"
server_key_iv: "94d4ece240903e0c4049ed6e279d439c"
server_key_version: "encryptionkey@0"
I20240507 02:07:21.191018 5487 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240507 02:07:21.191687 5502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:21.191839 5487 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240507 02:07:21.191915 5487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/minicluster-data/ts-0/wal
uuid: "0cd50473bac94e81beb0660cbc43d2be"
format_stamp: "Formatted at 2024-05-07 02:07:21 on dist-test-slave-v0bg"
server_key: "1c9ea1dc2a9195bc1c54eb23e6778dc6"
server_key_iv: "94d4ece240903e0c4049ed6e279d439c"
server_key_version: "encryptionkey@0"
I20240507 02:07:21.191982 5487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:07:21.207104 5487 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:07:21.208446 5487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:07:21.208528 5487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:07:21.208655 5487 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:07:21.208891 5487 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:07:21.208921 5487 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:21.208940 5487 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:07:21.208962 5487 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:21.213651 5487 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.65:37073
I20240507 02:07:21.213683 5616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.65:37073 every 8 connection(s)
I20240507 02:07:21.213915 5487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
I20240507 02:07:21.215637 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 5487
I20240507 02:07:21.215728 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/minicluster-data/ts-0/wal/instance
I20240507 02:07:21.215895 32269 external_mini_cluster.cc:1428] Setting key 36b48bf600bbbf96367ec109cc5da7ec
May 07 02:07:21 dist-test-slave-v0bg krb5kdc[5379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047641, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.65@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:07:21.221443 5483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:21.214461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:58499 (local address 127.31.131.126:35425)
0507 02:07:21.214510 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:21.214511 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:21.214661 (+ 150us) server_negotiation.cc:407] Connection header received
0507 02:07:21.215066 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:21.215067 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:21.215075 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:21.215114 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:21.215846 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:21.216334 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:21.216955 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:21.217067 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:21.219092 (+ 2025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:21.219096 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:21.219096 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:07:21.219103 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:21.220319 (+ 1216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:21.220734 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:21.220734 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:21.220735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:21.220751 (+ 16us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:21.220937 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:21.220938 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:21.220938 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:21.221009 (+ 71us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:21.221048 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:21.221315 (+ 267us) server_negotiation.cc:299] Negotiation successful
0507 02:07:21.221374 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20240507 02:07:21.221604 5619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:21.214421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:35425 (local address 127.31.131.65:58499)
0507 02:07:21.214565 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:21.214592 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:21.214982 (+ 390us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:21.215166 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:21.215169 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:21.215418 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:07:21.215767 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:21.215772 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:21.216397 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:21.216398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:21.216882 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:21.216885 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:21.217237 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:21.217646 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:07:21.217664 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:07:21.219000 (+ 1336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:07:21.220398 (+ 1398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:21.220405 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:21.220407 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:21.220662 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:21.220800 (+ 138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:21.220801 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:21.220802 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:21.220877 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:21.221056 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:07:21.221060 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:07:21.221193 (+ 133us) client_negotiation.cc:769] Sending connection context
0507 02:07:21.221312 (+ 119us) client_negotiation.cc:240] Negotiation successful
0507 02:07:21.221436 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":73,"threads_started":1}
I20240507 02:07:21.222079 5617 heartbeater.cc:347] Connected to a master server at 127.31.131.126:35425
I20240507 02:07:21.222155 5617 heartbeater.cc:464] Registering TS with master...
I20240507 02:07:21.222306 5617 heartbeater.cc:511] Master 127.31.131.126:35425 requested a full tablet report, sending...
I20240507 02:07:21.222774 5424 ts_manager.cc:194] Registered new tserver with Master: 0cd50473bac94e81beb0660cbc43d2be (127.31.131.65:37073)
WARNING: no policy specified for kudu/127.31.131.66@KRBTEST.COM; defaulting to no policy
I20240507 02:07:21.223371 5424 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.131.65@KRBTEST.COM'} at 127.31.131.65:58499
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab.
Entry for principal kudu/127.31.131.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab.
I20240507 02:07:21.236621 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.66:0
--local_ip_for_outbound_sockets=127.31.131.66
--webserver_interface=127.31.131.66
--webserver_port=0
--tserver_master_addrs=127.31.131.126:35425
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab
--principal=kudu/127.31.131.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:45965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:07:21.309206 5622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:07:21.309356 5622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:07:21.309382 5622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:07:21.310629 5622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:07:21.310667 5622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:07:21.310699 5622 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.66
I20240507 02:07:21.312183 5622 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:45965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab
--principal=kudu/127.31.131.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.131.66
--webserver_port=0
--tserver_master_addrs=127.31.131.126:35425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5622
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.66
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:07:21.312376 5622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:07:21.312536 5622 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:07:21.316102 5627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:07:21.316154 5622 server_base.cc:1034] running on GCE node
W20240507 02:07:21.316236 5628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:07:21.316293 5630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:07:21.316480 5622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:07:21.316674 5622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:07:21.317809 5622 hybrid_clock.cc:648] HybridClock initialized: now 1715047641317793 us; error 28 us; skew 500 ppm
May 07 02:07:21 dist-test-slave-v0bg krb5kdc[5379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047641, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:07:21.319989 5622 init.cc:377] Logged in from keytab as kudu/127.31.131.66@KRBTEST.COM (short username kudu)
I20240507 02:07:21.320356 5622 webserver.cc:460] Webserver started at http://127.31.131.66:38881/ using document root <none> and password file <none>
I20240507 02:07:21.320515 5622 fs_manager.cc:359] Metadata directory not provided
I20240507 02:07:21.320551 5622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:07:21.320637 5622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:07:21.321559 5622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/minicluster-data/ts-1/data/instance:
uuid: "ba24237972f5467f9ae92aafb1bd7712"
format_stamp: "Formatted at 2024-05-07 02:07:21 on dist-test-slave-v0bg"
server_key: "9d49155f7c60488c0fd8c88fc744995d"
server_key_iv: "7257771a7cf47e4ba0471e09c05855b0"
server_key_version: "encryptionkey@0"
I20240507 02:07:21.321835 5622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/minicluster-data/ts-1/wal/instance:
uuid: "ba24237972f5467f9ae92aafb1bd7712"
format_stamp: "Formatted at 2024-05-07 02:07:21 on dist-test-slave-v0bg"
server_key: "9d49155f7c60488c0fd8c88fc744995d"
server_key_iv: "7257771a7cf47e4ba0471e09c05855b0"
server_key_version: "encryptionkey@0"
I20240507 02:07:21.322922 5622 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240507 02:07:21.323518 5637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:21.323675 5622 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240507 02:07:21.323746 5622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/minicluster-data/ts-1/wal
uuid: "ba24237972f5467f9ae92aafb1bd7712"
format_stamp: "Formatted at 2024-05-07 02:07:21 on dist-test-slave-v0bg"
server_key: "9d49155f7c60488c0fd8c88fc744995d"
server_key_iv: "7257771a7cf47e4ba0471e09c05855b0"
server_key_version: "encryptionkey@0"
I20240507 02:07:21.323794 5622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:07:21.368921 5622 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:07:21.370471 5622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:07:21.370602 5622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:07:21.370776 5622 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:07:21.371064 5622 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:07:21.371096 5622 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:21.371119 5622 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:07:21.371141 5622 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:21.375795 5622 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.66:39585
I20240507 02:07:21.375865 5751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.66:39585 every 8 connection(s)
I20240507 02:07:21.376097 5622 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
I20240507 02:07:21.380012 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 5622
I20240507 02:07:21.380086 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/minicluster-data/ts-1/wal/instance
I20240507 02:07:21.380240 32269 external_mini_cluster.cc:1428] Setting key b7633f75564a62a625f2e2a5ed6eb377
May 07 02:07:21 dist-test-slave-v0bg krb5kdc[5379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047641, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.66@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:07:21.383668 5483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:21.376717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.66:60201 (local address 127.31.131.126:35425)
0507 02:07:21.376758 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:21.376760 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:21.376923 (+ 163us) server_negotiation.cc:407] Connection header received
0507 02:07:21.377383 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:21.377384 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:21.377396 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:21.377442 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:21.378141 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:21.378649 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:21.379165 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:21.379334 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:21.381315 (+ 1981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:21.381318 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:21.381319 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:07:21.381325 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:21.382660 (+ 1335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:21.382972 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:21.382973 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:21.382973 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:21.382997 (+ 24us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:21.383144 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:21.383145 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:21.383145 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:21.383230 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:21.383281 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:21.383495 (+ 214us) server_negotiation.cc:299] Negotiation successful
0507 02:07:21.383577 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20240507 02:07:21.383744 5754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:21.376692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:35425 (local address 127.31.131.66:60201)
0507 02:07:21.376824 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:21.376849 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:21.377287 (+ 438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:21.377508 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:21.377512 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:21.377775 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:07:21.378049 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:21.378052 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:21.378721 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:21.378722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:21.379090 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:21.379092 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:21.379434 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:21.379823 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:07:21.379835 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:07:21.381230 (+ 1395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:07:21.382725 (+ 1495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:21.382730 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:21.382732 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:21.382910 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:21.383046 (+ 136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:21.383047 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:21.383047 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:21.383097 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:21.383278 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:07:21.383282 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:07:21.383382 (+ 100us) client_negotiation.cc:769] Sending connection context
0507 02:07:21.383468 (+ 86us) client_negotiation.cc:240] Negotiation successful
0507 02:07:21.383579 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":58,"threads_started":1}
I20240507 02:07:21.384086 5752 heartbeater.cc:347] Connected to a master server at 127.31.131.126:35425
I20240507 02:07:21.384166 5752 heartbeater.cc:464] Registering TS with master...
I20240507 02:07:21.384328 5752 heartbeater.cc:511] Master 127.31.131.126:35425 requested a full tablet report, sending...
I20240507 02:07:21.384656 5424 ts_manager.cc:194] Registered new tserver with Master: ba24237972f5467f9ae92aafb1bd7712 (127.31.131.66:39585)
I20240507 02:07:21.385039 5424 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.131.66@KRBTEST.COM'} at 127.31.131.66:60201
WARNING: no policy specified for kudu/127.31.131.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab.
Entry for principal kudu/127.31.131.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab.
I20240507 02:07:21.400442 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.67:0
--local_ip_for_outbound_sockets=127.31.131.67
--webserver_interface=127.31.131.67
--webserver_port=0
--tserver_master_addrs=127.31.131.126:35425
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab
--principal=kudu/127.31.131.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:45965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:07:21.473207 5757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:07:21.473361 5757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:07:21.473379 5757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:07:21.474658 5757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:07:21.474694 5757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:07:21.474721 5757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.67
I20240507 02:07:21.476222 5757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:45965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab
--principal=kudu/127.31.131.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.131.67
--webserver_port=0
--tserver_master_addrs=127.31.131.126:35425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.67
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:07:21.476404 5757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:07:21.476569 5757 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:07:21.480134 5762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:07:21.480157 5765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:07:21.480295 5757 server_base.cc:1034] running on GCE node
W20240507 02:07:21.480325 5763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:07:21.480473 5757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:07:21.480652 5757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:07:21.481771 5757 hybrid_clock.cc:648] HybridClock initialized: now 1715047641481756 us; error 32 us; skew 500 ppm
May 07 02:07:21 dist-test-slave-v0bg krb5kdc[5379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047641, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:07:21.483879 5757 init.cc:377] Logged in from keytab as kudu/127.31.131.67@KRBTEST.COM (short username kudu)
I20240507 02:07:21.484247 5757 webserver.cc:460] Webserver started at http://127.31.131.67:40965/ using document root <none> and password file <none>
I20240507 02:07:21.484393 5757 fs_manager.cc:359] Metadata directory not provided
I20240507 02:07:21.484428 5757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:07:21.484525 5757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:07:21.485401 5757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/minicluster-data/ts-2/data/instance:
uuid: "f8fa19172a304167b169460ddc68085c"
format_stamp: "Formatted at 2024-05-07 02:07:21 on dist-test-slave-v0bg"
server_key: "235ddcdc408cb2c44671d993006dec11"
server_key_iv: "25dd2a2d249fd33c4c991b39f2c5762a"
server_key_version: "encryptionkey@0"
I20240507 02:07:21.485664 5757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/minicluster-data/ts-2/wal/instance:
uuid: "f8fa19172a304167b169460ddc68085c"
format_stamp: "Formatted at 2024-05-07 02:07:21 on dist-test-slave-v0bg"
server_key: "235ddcdc408cb2c44671d993006dec11"
server_key_iv: "25dd2a2d249fd33c4c991b39f2c5762a"
server_key_version: "encryptionkey@0"
I20240507 02:07:21.486678 5757 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240507 02:07:21.487244 5772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:21.487375 5757 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:21.487434 5757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/minicluster-data/ts-2/wal
uuid: "f8fa19172a304167b169460ddc68085c"
format_stamp: "Formatted at 2024-05-07 02:07:21 on dist-test-slave-v0bg"
server_key: "235ddcdc408cb2c44671d993006dec11"
server_key_iv: "25dd2a2d249fd33c4c991b39f2c5762a"
server_key_version: "encryptionkey@0"
I20240507 02:07:21.487491 5757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:07:21.496080 5757 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:07:21.497409 5757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:07:21.497495 5757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:07:21.497651 5757 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:07:21.497885 5757 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:07:21.497915 5757 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:21.497933 5757 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:07:21.497960 5757 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:21.502606 5757 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.67:39383
I20240507 02:07:21.502640 5885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.67:39383 every 8 connection(s)
I20240507 02:07:21.502877 5757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
I20240507 02:07:21.503254 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 5757
I20240507 02:07:21.503342 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/minicluster-data/ts-2/wal/instance
I20240507 02:07:21.503506 32269 external_mini_cluster.cc:1428] Setting key 0977f6f66aa698ee6c5bf3b92a47c63b
May 07 02:07:21 dist-test-slave-v0bg krb5kdc[5379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047641, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.67@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:07:21.509491 5483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:21.503485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.67:45599 (local address 127.31.131.126:35425)
0507 02:07:21.503532 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:21.503534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:21.503678 (+ 144us) server_negotiation.cc:407] Connection header received
0507 02:07:21.504094 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:21.504095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:21.504105 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:21.504160 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:21.504792 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:21.505260 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:21.505720 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:21.505818 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:21.507643 (+ 1825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:21.507647 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:21.507647 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:07:21.507653 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:21.508654 (+ 1001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:21.508943 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:21.508943 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:21.508944 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:21.508964 (+ 20us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:21.509082 (+ 118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:21.509082 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:21.509082 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:21.509158 (+ 76us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:21.509199 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:21.509369 (+ 170us) server_negotiation.cc:299] Negotiation successful
0507 02:07:21.509430 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20240507 02:07:21.509575 5888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:21.503416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:35425 (local address 127.31.131.67:45599)
0507 02:07:21.503565 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:21.503594 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:21.504026 (+ 432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:21.504201 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:21.504205 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:21.504468 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:07:21.504724 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:21.504728 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:21.505318 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:21.505318 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:21.505672 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:21.505673 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:21.505955 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:21.506362 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:07:21.506376 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:07:21.507568 (+ 1192us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:07:21.508712 (+ 1144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:21.508716 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:21.508718 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:21.508879 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:21.509005 (+ 126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:21.509005 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:21.509006 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:21.509046 (+ 40us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:21.509196 (+ 150us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:07:21.509198 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:07:21.509283 (+ 85us) client_negotiation.cc:769] Sending connection context
0507 02:07:21.509356 (+ 73us) client_negotiation.cc:240] Negotiation successful
0507 02:07:21.509442 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":75,"threads_started":1}
I20240507 02:07:21.509984 5886 heartbeater.cc:347] Connected to a master server at 127.31.131.126:35425
I20240507 02:07:21.510059 5886 heartbeater.cc:464] Registering TS with master...
I20240507 02:07:21.510226 5886 heartbeater.cc:511] Master 127.31.131.126:35425 requested a full tablet report, sending...
I20240507 02:07:21.510583 5424 ts_manager.cc:194] Registered new tserver with Master: f8fa19172a304167b169460ddc68085c (127.31.131.67:39383)
I20240507 02:07:21.510929 5424 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.131.67@KRBTEST.COM'} at 127.31.131.67:45599
I20240507 02:07:21.514984 32269 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/krb5kdc/test-admin.keytab.
May 07 02:07:21 dist-test-slave-v0bg krb5kdc[5379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:07:21.527248 32269 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20240507 02:07:21.528623 5893 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:07:21.527986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46970 (local address 127.31.131.65:37073)
0507 02:07:21.528141 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:21.528148 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:21.528156 (+ 8us) server_negotiation.cc:407] Connection header received
0507 02:07:21.528177 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:21.528179 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:21.528192 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:21.528252 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:21.528536 (+ 284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46970: BlockingRecv error: recv got EOF from 127.0.0.1:46970 (error 108)
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":88,"threads_started":1}
W20240507 02:07:21.528685 32269 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/krb5kdc/krb5cc' of type FILE
May 07 02:07:21 dist-test-slave-v0bg krb5kdc[5379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:07:21.529569 32269 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
May 07 02:07:21 dist-test-slave-v0bg krb5kdc[5379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:07:21.530320 32269 init.cc:295] Successfully reacquired a new kerberos TGT
May 07 02:07:21 dist-test-slave-v0bg krb5kdc[5379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.131.65@KRBTEST.COM
I20240507 02:07:21.539287 5893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:21.530828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46974 (local address 127.31.131.65:37073)
0507 02:07:21.530869 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:21.530871 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:21.530893 (+ 22us) server_negotiation.cc:407] Connection header received
0507 02:07:21.530945 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:21.530946 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:21.530955 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:21.531001 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:21.531456 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:21.532623 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:21.533388 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:21.533632 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:21.535655 (+ 2023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:21.535666 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:21.535666 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:07:21.535674 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:21.537723 (+ 2049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:21.538446 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:21.538450 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:21.538451 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:21.538500 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:21.538692 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:21.538693 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:21.538693 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:21.538808 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:21.538980 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:21.539089 (+ 109us) server_negotiation.cc:299] Negotiation successful
0507 02:07:21.539181 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20240507 02:07:21.541136 5893 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:07:21.540681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46978 (local address 127.31.131.65:37073)
0507 02:07:21.540730 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:21.540731 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:21.540739 (+ 8us) server_negotiation.cc:407] Connection header received
0507 02:07:21.540796 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:21.540796 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:21.540804 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:21.540841 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:21.541097 (+ 256us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46978: BlockingRecv error: recv got EOF from 127.0.0.1:46978 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20240507 02:07:21.541229 32269 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/krb5kdc/krb5cc' of type FILE
May 07 02:07:21 dist-test-slave-v0bg krb5kdc[5379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:07:21.542058 32269 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
May 07 02:07:21 dist-test-slave-v0bg krb5kdc[5379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:07:21.542842 32269 init.cc:295] Successfully reacquired a new kerberos TGT
May 07 02:07:21 dist-test-slave-v0bg krb5kdc[5379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.131.65@KRBTEST.COM
I20240507 02:07:21.548744 5893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:21.543314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46990 (local address 127.31.131.65:37073)
0507 02:07:21.543358 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:21.543360 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:21.543417 (+ 57us) server_negotiation.cc:407] Connection header received
0507 02:07:21.543474 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:21.543475 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:21.543483 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:21.543525 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:21.543937 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:21.544373 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:21.544851 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:21.545018 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:21.546527 (+ 1509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:21.546530 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:21.546531 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:07:21.546536 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:21.547855 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:21.548161 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:21.548162 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:21.548162 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:21.548184 (+ 22us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:21.548320 (+ 136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:21.548320 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:21.548321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:21.548394 (+ 73us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:21.548473 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:21.548585 (+ 112us) server_negotiation.cc:299] Negotiation successful
0507 02:07:21.548662 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
W20240507 02:07:21.549973 5893 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:07:21.549603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47004 (local address 127.31.131.65:37073)
0507 02:07:21.549645 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:21.549646 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:21.549654 (+ 8us) server_negotiation.cc:407] Connection header received
0507 02:07:21.549674 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:21.549674 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:21.549681 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:21.549730 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:21.549933 (+ 203us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47004: BlockingRecv error: recv got EOF from 127.0.0.1:47004 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20240507 02:07:21.550084 32269 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1715047604443657-32269-0/krb5kdc/krb5cc' of type FILE
May 07 02:07:21 dist-test-slave-v0bg krb5kdc[5379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:07:21.550927 32269 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
May 07 02:07:21 dist-test-slave-v0bg krb5kdc[5379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:07:21.551632 32269 init.cc:295] Successfully reacquired a new kerberos TGT
May 07 02:07:21 dist-test-slave-v0bg krb5kdc[5379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.131.65@KRBTEST.COM
I20240507 02:07:21.556881 5893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:21.552103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47020 (local address 127.31.131.65:37073)
0507 02:07:21.552142 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:21.552144 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:21.552152 (+ 8us) server_negotiation.cc:407] Connection header received
0507 02:07:21.552167 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:21.552168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:21.552174 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:21.552224 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:21.552600 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:21.553003 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:21.553434 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:21.553540 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:21.555061 (+ 1521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:21.555063 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:21.555064 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:07:21.555069 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:21.556250 (+ 1181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:21.556482 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:21.556482 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:21.556483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:21.556498 (+ 15us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:21.556590 (+ 92us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:21.556590 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:21.556590 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:21.556646 (+ 56us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:21.556705 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:21.556776 (+ 71us) server_negotiation.cc:299] Negotiation successful
0507 02:07:21.556829 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20240507 02:07:21.557451 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 5487
I20240507 02:07:21.562003 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 5622
I20240507 02:07:21.565632 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 5757
I20240507 02:07:21.569428 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 5392
2024-05-07T02:07:21Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1609 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 07 02:07:21 dist-test-slave-v0bg krb5kdc[5906](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 07 02:07:21 dist-test-slave-v0bg krb5kdc[5906](info): set up 2 sockets
May 07 02:07:21 dist-test-slave-v0bg krb5kdc[5906](info): commencing operation
krb5kdc: starting...
W20240507 02:07:22.455379 32269 mini_kdc.cc:121] Time spent starting KDC: real 0.868s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/krb5kdc/test-user.keytab.
May 07 02:07:22 dist-test-slave-v0bg krb5kdc[5906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-07T02:07:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-07T02:07:22Z Disabled control of system clock
WARNING: no policy specified for oryx/127.31.131.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.131.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.131.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/krb5kdc/oryx_127.31.131.126.keytab.
Entry for principal oryx/127.31.131.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/krb5kdc/oryx_127.31.131.126.keytab.
I20240507 02:07:22.555191 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:40295
--webserver_interface=127.31.131.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/krb5kdc/oryx_127.31.131.126.keytab
--principal=oryx/127.31.131.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:42939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.131.126:40295
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:07:22.627871 5919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:07:22.628028 5919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:07:22.628047 5919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:07:22.629251 5919 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240507 02:07:22.629292 5919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:07:22.629302 5919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:07:22.629315 5919 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240507 02:07:22.629338 5919 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240507 02:07:22.630764 5919 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:42939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.131.126:40295
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/krb5kdc/oryx_127.31.131.126.keytab
--principal=oryx/127.31.131.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:40295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.131.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5919
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:07:22.630939 5919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:07:22.631103 5919 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:07:22.634596 5925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:07:22.634586 5927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:07:22.634677 5919 server_base.cc:1034] running on GCE node
W20240507 02:07:22.634876 5924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:07:22.635087 5919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:07:22.635303 5919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:07:22.636438 5919 hybrid_clock.cc:648] HybridClock initialized: now 1715047642636422 us; error 33 us; skew 500 ppm
May 07 02:07:22 dist-test-slave-v0bg krb5kdc[5906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047642, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.131.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:07:22.638515 5919 init.cc:377] Logged in from keytab as oryx/127.31.131.126@KRBTEST.COM (short username oryx)
I20240507 02:07:22.638875 5919 webserver.cc:460] Webserver started at http://127.31.131.126:36625/ using document root <none> and password file <none>
I20240507 02:07:22.639020 5919 fs_manager.cc:359] Metadata directory not provided
I20240507 02:07:22.639052 5919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:07:22.639127 5919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:07:22.639956 5919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/master-0/data/instance:
uuid: "42c4002f2beb495880b7c5db3cabab77"
format_stamp: "Formatted at 2024-05-07 02:07:22 on dist-test-slave-v0bg"
server_key: "cc420455c3fd057e734a727729702b9b"
server_key_iv: "aaceeec667f2f10ed424c009f96d941b"
server_key_version: "encryptionkey@0"
I20240507 02:07:22.640188 5919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/master-0/wal/instance:
uuid: "42c4002f2beb495880b7c5db3cabab77"
format_stamp: "Formatted at 2024-05-07 02:07:22 on dist-test-slave-v0bg"
server_key: "cc420455c3fd057e734a727729702b9b"
server_key_iv: "aaceeec667f2f10ed424c009f96d941b"
server_key_version: "encryptionkey@0"
I20240507 02:07:22.641168 5919 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240507 02:07:22.641688 5934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:22.641822 5919 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240507 02:07:22.641888 5919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/master-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/master-0/wal
uuid: "42c4002f2beb495880b7c5db3cabab77"
format_stamp: "Formatted at 2024-05-07 02:07:22 on dist-test-slave-v0bg"
server_key: "cc420455c3fd057e734a727729702b9b"
server_key_iv: "aaceeec667f2f10ed424c009f96d941b"
server_key_version: "encryptionkey@0"
I20240507 02:07:22.641953 5919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:07:22.670094 5919 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:07:22.671695 5919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:07:22.671798 5919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:07:22.674934 5919 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.126:40295
I20240507 02:07:22.674968 5986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.126:40295 every 8 connection(s)
I20240507 02:07:22.675248 5919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
I20240507 02:07:22.675982 5987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:07:22.677809 5987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42c4002f2beb495880b7c5db3cabab77: Bootstrap starting.
I20240507 02:07:22.678251 5987 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42c4002f2beb495880b7c5db3cabab77: Neither blocks nor log segments found. Creating new log.
I20240507 02:07:22.678303 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 5919
I20240507 02:07:22.678397 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/master-0/wal/instance
I20240507 02:07:22.678490 5987 log.cc:824] T 00000000000000000000000000000000 P 42c4002f2beb495880b7c5db3cabab77: Log is configured to *not* fsync() on all Append() calls
I20240507 02:07:22.678539 32269 external_mini_cluster.cc:1428] Setting key e6682e7fe9d72f545960585d035a01b1
I20240507 02:07:22.679147 5987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42c4002f2beb495880b7c5db3cabab77: No bootstrap required, opened a new log
I20240507 02:07:22.680508 5987 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42c4002f2beb495880b7c5db3cabab77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42c4002f2beb495880b7c5db3cabab77" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 40295 } }
I20240507 02:07:22.680614 5987 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42c4002f2beb495880b7c5db3cabab77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:07:22.680636 5987 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42c4002f2beb495880b7c5db3cabab77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42c4002f2beb495880b7c5db3cabab77, State: Initialized, Role: FOLLOWER
I20240507 02:07:22.680720 5987 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42c4002f2beb495880b7c5db3cabab77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42c4002f2beb495880b7c5db3cabab77" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 40295 } }
I20240507 02:07:22.680778 5987 raft_consensus.cc:388] T 00000000000000000000000000000000 P 42c4002f2beb495880b7c5db3cabab77 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 02:07:22.680804 5987 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42c4002f2beb495880b7c5db3cabab77 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 02:07:22.680845 5987 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42c4002f2beb495880b7c5db3cabab77 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:07:22.681283 5987 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42c4002f2beb495880b7c5db3cabab77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42c4002f2beb495880b7c5db3cabab77" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 40295 } }
I20240507 02:07:22.681366 5987 leader_election.cc:302] T 00000000000000000000000000000000 P 42c4002f2beb495880b7c5db3cabab77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42c4002f2beb495880b7c5db3cabab77; no voters:
I20240507 02:07:22.681486 5987 leader_election.cc:288] T 00000000000000000000000000000000 P 42c4002f2beb495880b7c5db3cabab77 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240507 02:07:22.681560 5992 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42c4002f2beb495880b7c5db3cabab77 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:07:22.681689 5992 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42c4002f2beb495880b7c5db3cabab77 [term 1 LEADER]: Becoming Leader. State: Replica: 42c4002f2beb495880b7c5db3cabab77, State: Running, Role: LEADER
I20240507 02:07:22.681710 5987 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42c4002f2beb495880b7c5db3cabab77 [sys.catalog]: configured and running, proceeding with master startup.
I20240507 02:07:22.681854 5992 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42c4002f2beb495880b7c5db3cabab77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42c4002f2beb495880b7c5db3cabab77" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 40295 } }
I20240507 02:07:22.682781 5993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42c4002f2beb495880b7c5db3cabab77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42c4002f2beb495880b7c5db3cabab77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42c4002f2beb495880b7c5db3cabab77" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 40295 } } }
I20240507 02:07:22.682859 5993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42c4002f2beb495880b7c5db3cabab77 [sys.catalog]: This master's current role is: LEADER
I20240507 02:07:22.683043 6005 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 02:07:22.683178 5994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42c4002f2beb495880b7c5db3cabab77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42c4002f2beb495880b7c5db3cabab77. Latest consensus state: current_term: 1 leader_uuid: "42c4002f2beb495880b7c5db3cabab77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42c4002f2beb495880b7c5db3cabab77" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 40295 } } }
I20240507 02:07:22.683333 5994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42c4002f2beb495880b7c5db3cabab77 [sys.catalog]: This master's current role is: LEADER
I20240507 02:07:22.683550 6005 catalog_manager.cc:1484] Initializing Kudu cluster ID...
May 07 02:07:22 dist-test-slave-v0bg krb5kdc[5906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM
I20240507 02:07:22.684787 6005 catalog_manager.cc:1347] Generated new cluster ID: 4b358b1693354b3488e6edbc34acd7c3
I20240507 02:07:22.684834 6005 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240507 02:07:22.688201 5991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:22.678978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49740 (local address 127.31.131.126:40295)
0507 02:07:22.679128 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:22.679139 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:22.679156 (+ 17us) server_negotiation.cc:407] Connection header received
0507 02:07:22.679578 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:22.679585 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:22.679624 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:22.679739 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:22.680442 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:22.681473 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:22.682195 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:22.682781 (+ 586us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:22.685158 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:22.685163 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:22.685163 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:07:22.685179 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:22.686991 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:22.687326 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:22.687328 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:22.687328 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:22.687373 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:22.687515 (+ 142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:22.687516 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:22.687516 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:22.687700 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:22.687820 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:22.687928 (+ 108us) server_negotiation.cc:299] Negotiation successful
0507 02:07:22.688026 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":71,"threads_started":1}
I20240507 02:07:22.706166 6005 catalog_manager.cc:1370] Generated new certificate authority record
I20240507 02:07:22.706669 6005 catalog_manager.cc:1504] Loading token signing keys...
I20240507 02:07:22.711232 6005 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 42c4002f2beb495880b7c5db3cabab77: Generated new TSK 0
I20240507 02:07:22.711361 6005 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.31.131.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.131.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.131.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/krb5kdc/oryx_127.31.131.65.keytab.
Entry for principal oryx/127.31.131.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/krb5kdc/oryx_127.31.131.65.keytab.
I20240507 02:07:22.738906 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.65:0
--local_ip_for_outbound_sockets=127.31.131.65
--webserver_interface=127.31.131.65
--webserver_port=0
--tserver_master_addrs=127.31.131.126:40295
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/krb5kdc/oryx_127.31.131.65.keytab
--principal=oryx/127.31.131.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:42939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:07:22.814327 6013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:07:22.814513 6013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:07:22.814571 6013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:07:22.815822 6013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:07:22.815876 6013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:07:22.815912 6013 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.65
I20240507 02:07:22.817430 6013 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:42939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/krb5kdc/oryx_127.31.131.65.keytab
--principal=oryx/127.31.131.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.131.65
--webserver_port=0
--tserver_master_addrs=127.31.131.126:40295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.65
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:07:22.817642 6013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:07:22.817811 6013 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:07:22.821872 6019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:07:22.821897 6018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:07:22.822039 6013 server_base.cc:1034] running on GCE node
W20240507 02:07:22.822180 6021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:07:22.822345 6013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:07:22.822543 6013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:07:22.823719 6013 hybrid_clock.cc:648] HybridClock initialized: now 1715047642823704 us; error 34 us; skew 500 ppm
May 07 02:07:22 dist-test-slave-v0bg krb5kdc[5906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047642, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.131.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:07:22.825930 6013 init.cc:377] Logged in from keytab as oryx/127.31.131.65@KRBTEST.COM (short username oryx)
I20240507 02:07:22.826263 6013 webserver.cc:460] Webserver started at http://127.31.131.65:45227/ using document root <none> and password file <none>
I20240507 02:07:22.826395 6013 fs_manager.cc:359] Metadata directory not provided
I20240507 02:07:22.826435 6013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:07:22.826510 6013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:07:22.827418 6013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-0/data/instance:
uuid: "081c850177844c7696632cea1ad6eb8a"
format_stamp: "Formatted at 2024-05-07 02:07:22 on dist-test-slave-v0bg"
server_key: "747fb2588ffe186983dbb85df38bb853"
server_key_iv: "1cffad56ba804b397e9afd72d1efaae7"
server_key_version: "encryptionkey@0"
I20240507 02:07:22.827669 6013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-0/wal/instance:
uuid: "081c850177844c7696632cea1ad6eb8a"
format_stamp: "Formatted at 2024-05-07 02:07:22 on dist-test-slave-v0bg"
server_key: "747fb2588ffe186983dbb85df38bb853"
server_key_iv: "1cffad56ba804b397e9afd72d1efaae7"
server_key_version: "encryptionkey@0"
I20240507 02:07:22.828667 6013 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240507 02:07:22.829217 6028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:22.829386 6013 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240507 02:07:22.829455 6013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-0/wal
uuid: "081c850177844c7696632cea1ad6eb8a"
format_stamp: "Formatted at 2024-05-07 02:07:22 on dist-test-slave-v0bg"
server_key: "747fb2588ffe186983dbb85df38bb853"
server_key_iv: "1cffad56ba804b397e9afd72d1efaae7"
server_key_version: "encryptionkey@0"
I20240507 02:07:22.829507 6013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:07:22.852185 6013 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:07:22.853639 6013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:07:22.853729 6013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:07:22.853891 6013 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:07:22.854125 6013 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:07:22.854154 6013 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:22.854176 6013 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:07:22.854197 6013 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:22.859696 6013 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.65:38131
I20240507 02:07:22.859752 6142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.65:38131 every 8 connection(s)
I20240507 02:07:22.860008 6013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
I20240507 02:07:22.862404 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 6013
I20240507 02:07:22.862501 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-0/wal/instance
I20240507 02:07:22.862672 32269 external_mini_cluster.cc:1428] Setting key 5e559872a5d43243a9f19277d9a19279
May 07 02:07:22 dist-test-slave-v0bg krb5kdc[5906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047642, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.131.65@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM
I20240507 02:07:22.868099 5991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:22.860647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:53165 (local address 127.31.131.126:40295)
0507 02:07:22.860703 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:22.860705 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:22.860842 (+ 137us) server_negotiation.cc:407] Connection header received
0507 02:07:22.861281 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:22.861282 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:22.861292 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:22.861352 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:22.862264 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:22.862832 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:22.863460 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:22.863583 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:22.865584 (+ 2001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:22.865587 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:22.865588 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:07:22.865593 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:22.866893 (+ 1300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:22.867270 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:22.867271 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:22.867272 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:22.867300 (+ 28us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:22.867530 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:22.867530 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:22.867531 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:22.867641 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:22.867708 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:22.867917 (+ 209us) server_negotiation.cc:299] Negotiation successful
0507 02:07:22.868007 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20240507 02:07:22.868108 6145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:22.860607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.65:53165)
0507 02:07:22.860740 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:22.860768 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:22.861198 (+ 430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:22.861400 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:22.861404 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:22.861654 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:07:22.862193 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:22.862197 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:22.862923 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:22.862924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:22.863382 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:22.863384 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:22.863514 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:22.863967 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:07:22.863985 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:07:22.865477 (+ 1492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:07:22.866981 (+ 1504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:22.866986 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:22.866988 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:22.867185 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:22.867386 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:22.867387 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:22.867387 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:22.867464 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:22.867701 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:07:22.867704 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:07:22.867799 (+ 95us) client_negotiation.cc:769] Sending connection context
0507 02:07:22.867883 (+ 84us) client_negotiation.cc:240] Negotiation successful
0507 02:07:22.867980 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":64,"threads_started":1}
I20240507 02:07:22.868683 6143 heartbeater.cc:347] Connected to a master server at 127.31.131.126:40295
I20240507 02:07:22.868770 6143 heartbeater.cc:464] Registering TS with master...
I20240507 02:07:22.868953 6143 heartbeater.cc:511] Master 127.31.131.126:40295 requested a full tablet report, sending...
I20240507 02:07:22.869474 5951 ts_manager.cc:194] Registered new tserver with Master: 081c850177844c7696632cea1ad6eb8a (127.31.131.65:38131)
I20240507 02:07:22.870390 5951 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.131.65@KRBTEST.COM'} at 127.31.131.65:53165
WARNING: no policy specified for oryx/127.31.131.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.131.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.131.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/krb5kdc/oryx_127.31.131.66.keytab.
Entry for principal oryx/127.31.131.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/krb5kdc/oryx_127.31.131.66.keytab.
I20240507 02:07:22.885248 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.66:0
--local_ip_for_outbound_sockets=127.31.131.66
--webserver_interface=127.31.131.66
--webserver_port=0
--tserver_master_addrs=127.31.131.126:40295
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/krb5kdc/oryx_127.31.131.66.keytab
--principal=oryx/127.31.131.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:42939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:07:22.958441 6148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:07:22.958612 6148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:07:22.958632 6148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:07:22.959825 6148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:07:22.959859 6148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:07:22.959882 6148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.66
I20240507 02:07:22.961230 6148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:42939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/krb5kdc/oryx_127.31.131.66.keytab
--principal=oryx/127.31.131.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.131.66
--webserver_port=0
--tserver_master_addrs=127.31.131.126:40295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.66
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:07:22.961380 6148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:07:22.961503 6148 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:07:22.965076 6156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:07:22.965173 6148 server_base.cc:1034] running on GCE node
W20240507 02:07:22.965222 6154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:07:22.965076 6153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:07:22.965438 6148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:07:22.965631 6148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:07:22.966758 6148 hybrid_clock.cc:648] HybridClock initialized: now 1715047642966738 us; error 33 us; skew 500 ppm
May 07 02:07:22 dist-test-slave-v0bg krb5kdc[5906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047642, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.131.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:07:22.968963 6148 init.cc:377] Logged in from keytab as oryx/127.31.131.66@KRBTEST.COM (short username oryx)
I20240507 02:07:22.969333 6148 webserver.cc:460] Webserver started at http://127.31.131.66:36661/ using document root <none> and password file <none>
I20240507 02:07:22.969481 6148 fs_manager.cc:359] Metadata directory not provided
I20240507 02:07:22.969519 6148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:07:22.969614 6148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:07:22.970479 6148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-1/data/instance:
uuid: "9bebe757b6734612ad65a04dd82d54f1"
format_stamp: "Formatted at 2024-05-07 02:07:22 on dist-test-slave-v0bg"
server_key: "518516610fc8c1009309e2066d4c5684"
server_key_iv: "020d854e9f9062a57befb61c36a515ef"
server_key_version: "encryptionkey@0"
I20240507 02:07:22.970788 6148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-1/wal/instance:
uuid: "9bebe757b6734612ad65a04dd82d54f1"
format_stamp: "Formatted at 2024-05-07 02:07:22 on dist-test-slave-v0bg"
server_key: "518516610fc8c1009309e2066d4c5684"
server_key_iv: "020d854e9f9062a57befb61c36a515ef"
server_key_version: "encryptionkey@0"
I20240507 02:07:22.971787 6148 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240507 02:07:22.972342 6163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:22.972477 6148 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:22.972543 6148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-1/wal
uuid: "9bebe757b6734612ad65a04dd82d54f1"
format_stamp: "Formatted at 2024-05-07 02:07:22 on dist-test-slave-v0bg"
server_key: "518516610fc8c1009309e2066d4c5684"
server_key_iv: "020d854e9f9062a57befb61c36a515ef"
server_key_version: "encryptionkey@0"
I20240507 02:07:22.972596 6148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:07:22.991482 6148 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:07:22.992857 6148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:07:22.992966 6148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:07:22.993130 6148 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:07:22.993387 6148 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:07:22.993417 6148 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:22.993443 6148 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:07:22.993458 6148 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:22.998175 6148 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.66:37593
I20240507 02:07:22.998222 6276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.66:37593 every 8 connection(s)
I20240507 02:07:22.998476 6148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
May 07 02:07:23 dist-test-slave-v0bg krb5kdc[5906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047642, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.131.66@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM
I20240507 02:07:23.005471 5991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:22.999109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.66:48177 (local address 127.31.131.126:40295)
0507 02:07:22.999170 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:22.999173 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:22.999296 (+ 123us) server_negotiation.cc:407] Connection header received
0507 02:07:22.999735 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:22.999736 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:22.999747 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:22.999795 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:23.000668 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.001148 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:23.001647 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.001806 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:23.003422 (+ 1616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:23.003425 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:23.003426 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:07:23.003432 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:23.004476 (+ 1044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:23.004784 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:23.004786 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:23.004786 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:23.004808 (+ 22us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:23.004958 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:23.004959 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:23.004959 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:23.005038 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:23.005091 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:23.005301 (+ 210us) server_negotiation.cc:299] Negotiation successful
0507 02:07:23.005380 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20240507 02:07:23.005564 6279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:22.999066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.66:48177)
0507 02:07:22.999206 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:22.999229 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:22.999645 (+ 416us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:22.999847 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:22.999850 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:23.000104 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:07:23.000578 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:23.000582 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.001220 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:23.001222 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:23.001576 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:23.001577 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.001693 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:23.002119 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:07:23.002132 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:07:23.003341 (+ 1209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:07:23.004547 (+ 1206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:23.004552 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:23.004555 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:23.004725 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:23.004857 (+ 132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:23.004858 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:23.004858 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:23.004907 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:23.005088 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:07:23.005092 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:07:23.005190 (+ 98us) client_negotiation.cc:769] Sending connection context
0507 02:07:23.005276 (+ 86us) client_negotiation.cc:240] Negotiation successful
0507 02:07:23.005387 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":70,"threads_started":1}
I20240507 02:07:23.005975 6277 heartbeater.cc:347] Connected to a master server at 127.31.131.126:40295
I20240507 02:07:23.006052 6277 heartbeater.cc:464] Registering TS with master...
I20240507 02:07:23.006204 6277 heartbeater.cc:511] Master 127.31.131.126:40295 requested a full tablet report, sending...
I20240507 02:07:23.006536 5951 ts_manager.cc:194] Registered new tserver with Master: 9bebe757b6734612ad65a04dd82d54f1 (127.31.131.66:37593)
I20240507 02:07:23.006899 5951 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.131.66@KRBTEST.COM'} at 127.31.131.66:48177
I20240507 02:07:23.008682 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 6148
I20240507 02:07:23.008772 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-1/wal/instance
I20240507 02:07:23.008926 32269 external_mini_cluster.cc:1428] Setting key 7baf3c4b25e2eb2ab923c82c47667cae
WARNING: no policy specified for oryx/127.31.131.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.131.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.131.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/krb5kdc/oryx_127.31.131.67.keytab.
Entry for principal oryx/127.31.131.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/krb5kdc/oryx_127.31.131.67.keytab.
I20240507 02:07:23.029126 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.67:0
--local_ip_for_outbound_sockets=127.31.131.67
--webserver_interface=127.31.131.67
--webserver_port=0
--tserver_master_addrs=127.31.131.126:40295
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/krb5kdc/oryx_127.31.131.67.keytab
--principal=oryx/127.31.131.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:42939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:07:23.104167 6282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:07:23.104348 6282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:07:23.104368 6282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:07:23.105602 6282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:07:23.105648 6282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:07:23.105674 6282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.67
I20240507 02:07:23.107125 6282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:42939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/krb5kdc/oryx_127.31.131.67.keytab
--principal=oryx/127.31.131.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.131.67
--webserver_port=0
--tserver_master_addrs=127.31.131.126:40295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6282
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.67
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:07:23.107313 6282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:07:23.107481 6282 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:07:23.111111 6290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:07:23.111212 6282 server_base.cc:1034] running on GCE node
W20240507 02:07:23.111240 6288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:07:23.111120 6287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:07:23.111428 6282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:07:23.111598 6282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:07:23.112711 6282 hybrid_clock.cc:648] HybridClock initialized: now 1715047643112697 us; error 33 us; skew 500 ppm
May 07 02:07:23 dist-test-slave-v0bg krb5kdc[5906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047643, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.131.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:07:23.114881 6282 init.cc:377] Logged in from keytab as oryx/127.31.131.67@KRBTEST.COM (short username oryx)
I20240507 02:07:23.115245 6282 webserver.cc:460] Webserver started at http://127.31.131.67:36947/ using document root <none> and password file <none>
I20240507 02:07:23.115422 6282 fs_manager.cc:359] Metadata directory not provided
I20240507 02:07:23.115458 6282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:07:23.115550 6282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:07:23.116451 6282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-2/data/instance:
uuid: "a94723bb3109427881c73d6e66034478"
format_stamp: "Formatted at 2024-05-07 02:07:23 on dist-test-slave-v0bg"
server_key: "91d58b57379293f373856e2e8b6ac62d"
server_key_iv: "c043f9fa4e35333a63c475494d0e479a"
server_key_version: "encryptionkey@0"
I20240507 02:07:23.116714 6282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-2/wal/instance:
uuid: "a94723bb3109427881c73d6e66034478"
format_stamp: "Formatted at 2024-05-07 02:07:23 on dist-test-slave-v0bg"
server_key: "91d58b57379293f373856e2e8b6ac62d"
server_key_iv: "c043f9fa4e35333a63c475494d0e479a"
server_key_version: "encryptionkey@0"
I20240507 02:07:23.117753 6282 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20240507 02:07:23.118306 6297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:23.118464 6282 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:23.118533 6282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-2/wal
uuid: "a94723bb3109427881c73d6e66034478"
format_stamp: "Formatted at 2024-05-07 02:07:23 on dist-test-slave-v0bg"
server_key: "91d58b57379293f373856e2e8b6ac62d"
server_key_iv: "c043f9fa4e35333a63c475494d0e479a"
server_key_version: "encryptionkey@0"
I20240507 02:07:23.118609 6282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:07:23.138273 6282 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:07:23.139773 6282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:07:23.139880 6282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:07:23.140056 6282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:07:23.140323 6282 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:07:23.140353 6282 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:23.140381 6282 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:07:23.140412 6282 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:23.145408 6282 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.67:41427
I20240507 02:07:23.145450 6410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.67:41427 every 8 connection(s)
I20240507 02:07:23.145718 6282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
May 07 02:07:23 dist-test-slave-v0bg krb5kdc[5906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047643, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.131.67@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM
I20240507 02:07:23.152608 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 6282
I20240507 02:07:23.152711 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-2/wal/instance
I20240507 02:07:23.152807 5991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:23.146317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.67:51751 (local address 127.31.131.126:40295)
0507 02:07:23.146379 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:23.146381 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:23.146543 (+ 162us) server_negotiation.cc:407] Connection header received
0507 02:07:23.147019 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:23.147020 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:23.147028 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:23.147079 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:23.147999 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.148468 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:23.148942 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.149039 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:23.150805 (+ 1766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:23.150809 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:23.150809 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:07:23.150815 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:23.151855 (+ 1040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:23.152156 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:23.152157 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:23.152157 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:23.152171 (+ 14us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:23.152295 (+ 124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:23.152295 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:23.152296 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:23.152373 (+ 77us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:23.152423 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:23.152643 (+ 220us) server_negotiation.cc:299] Negotiation successful
0507 02:07:23.152724 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20240507 02:07:23.152879 32269 external_mini_cluster.cc:1428] Setting key bbffa17d1db8b9d959af4404a140ec07
I20240507 02:07:23.152880 6413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:23.146298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.67:51751)
0507 02:07:23.146443 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:23.146472 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:23.146937 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:23.147125 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:23.147128 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:23.147418 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:07:23.147928 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:23.147931 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.148527 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:23.148528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:23.148889 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:23.148890 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.148984 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:23.149403 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:07:23.149415 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:07:23.150724 (+ 1309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:07:23.151918 (+ 1194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:23.151923 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:23.151926 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:23.152102 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:23.152213 (+ 111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:23.152213 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:23.152214 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:23.152258 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:23.152411 (+ 153us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:07:23.152413 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:07:23.152530 (+ 117us) client_negotiation.cc:769] Sending connection context
0507 02:07:23.152608 (+ 78us) client_negotiation.cc:240] Negotiation successful
0507 02:07:23.152713 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":63,"threads_started":1}
I20240507 02:07:23.153290 6411 heartbeater.cc:347] Connected to a master server at 127.31.131.126:40295
I20240507 02:07:23.153386 6411 heartbeater.cc:464] Registering TS with master...
I20240507 02:07:23.153584 6411 heartbeater.cc:511] Master 127.31.131.126:40295 requested a full tablet report, sending...
I20240507 02:07:23.153916 5950 ts_manager.cc:194] Registered new tserver with Master: a94723bb3109427881c73d6e66034478 (127.31.131.67:41427)
I20240507 02:07:23.154300 5950 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.131.67@KRBTEST.COM'} at 127.31.131.67:51751
I20240507 02:07:23.164816 32269 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.31.131.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.131.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.131.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/krb5kdc/oryx_127.31.131.125.keytab.
Entry for principal oryx/127.31.131.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/krb5kdc/oryx_127.31.131.125.keytab.
I20240507 02:07:23.185398 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.125:46259
--webserver_interface=127.31.131.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/krb5kdc/oryx_127.31.131.125.keytab
--principal=oryx/127.31.131.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:42939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.131.126:40295,127.31.131.125:46259
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:07:23.259907 6416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:07:23.260083 6416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:07:23.260111 6416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:07:23.261346 6416 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240507 02:07:23.261389 6416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:07:23.261406 6416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:07:23.261421 6416 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240507 02:07:23.261435 6416 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240507 02:07:23.262861 6416 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:42939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.31.131.126:40295,127.31.131.125:46259
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/krb5kdc/oryx_127.31.131.125.keytab
--principal=oryx/127.31.131.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.125:46259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.31.131.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
W20240507 02:07:23.262956 6416 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.31.131.126:40295,127.31.131.125:46259'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20240507 02:07:23.263078 6416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:07:23.263238 6416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:07:23.266808 6424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:07:23.266860 6416 server_base.cc:1034] running on GCE node
W20240507 02:07:23.266961 6421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:07:23.267073 6422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:07:23.267278 6416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:07:23.267496 6416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:07:23.268615 6416 hybrid_clock.cc:648] HybridClock initialized: now 1715047643268596 us; error 33 us; skew 500 ppm
May 07 02:07:23 dist-test-slave-v0bg krb5kdc[5906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047643, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.131.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:07:23.270798 6416 init.cc:377] Logged in from keytab as oryx/127.31.131.125@KRBTEST.COM (short username oryx)
I20240507 02:07:23.271183 6416 webserver.cc:460] Webserver started at http://127.31.131.125:40689/ using document root <none> and password file <none>
I20240507 02:07:23.271327 6416 fs_manager.cc:359] Metadata directory not provided
I20240507 02:07:23.271359 6416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:07:23.271436 6416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:07:23.272315 6416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/master-1/data/instance:
uuid: "063c163f1a014ed18bce90c55921d3ec"
format_stamp: "Formatted at 2024-05-07 02:07:23 on dist-test-slave-v0bg"
server_key: "799338019017974c19185ad6bc6e79b0"
server_key_iv: "3cd8d8bea26b09988b968aca56f239b1"
server_key_version: "encryptionkey@0"
I20240507 02:07:23.272552 6416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/master-1/wal/instance:
uuid: "063c163f1a014ed18bce90c55921d3ec"
format_stamp: "Formatted at 2024-05-07 02:07:23 on dist-test-slave-v0bg"
server_key: "799338019017974c19185ad6bc6e79b0"
server_key_iv: "3cd8d8bea26b09988b968aca56f239b1"
server_key_version: "encryptionkey@0"
I20240507 02:07:23.273525 6416 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20240507 02:07:23.274084 6431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:23.274236 6416 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:23.274322 6416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/master-1/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/master-1/wal
uuid: "063c163f1a014ed18bce90c55921d3ec"
format_stamp: "Formatted at 2024-05-07 02:07:23 on dist-test-slave-v0bg"
server_key: "799338019017974c19185ad6bc6e79b0"
server_key_iv: "3cd8d8bea26b09988b968aca56f239b1"
server_key_version: "encryptionkey@0"
I20240507 02:07:23.274382 6416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:07:23.296054 6416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:07:23.297871 6416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:07:23.297966 6416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:07:23.301048 6416 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.125:46259
I20240507 02:07:23.301080 6483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.125:46259 every 8 connection(s)
I20240507 02:07:23.301342 6416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/master-1/data/info.pb
I20240507 02:07:23.302060 6484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:07:23.303340 6484 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.131.126" port: 40295 } has no permanent_uuid. Determining permanent_uuid...
May 07 02:07:23 dist-test-slave-v0bg krb5kdc[5906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047643, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.131.125@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM
I20240507 02:07:23.308838 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 6416
I20240507 02:07:23.310182 5991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:23.303931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49752 (local address 127.31.131.126:40295)
0507 02:07:23.303988 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:23.303990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:23.304122 (+ 132us) server_negotiation.cc:407] Connection header received
0507 02:07:23.304547 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:23.304548 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:23.304557 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:23.304611 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:23.305222 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.305685 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:23.306162 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.306262 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:23.307860 (+ 1598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:23.307864 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:23.307864 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:07:23.307870 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:23.308948 (+ 1078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:23.309366 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:23.309368 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:23.309368 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:23.309399 (+ 31us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:23.309592 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:23.309592 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:23.309593 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:23.309742 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:23.309817 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:23.310014 (+ 197us) server_negotiation.cc:299] Negotiation successful
0507 02:07:23.310097 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20240507 02:07:23.310185 6486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:23.303869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.0.0.1:49752)
0507 02:07:23.304028 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:23.304054 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:23.304470 (+ 416us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:23.304661 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:23.304664 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:23.304907 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:07:23.305158 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:23.305162 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.305743 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:23.305744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:23.306108 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:23.306109 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.306194 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:23.306597 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:07:23.306609 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:07:23.307780 (+ 1171us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:07:23.309013 (+ 1233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:23.309018 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:23.309021 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:23.309265 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:23.309467 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:23.309468 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:23.309469 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:23.309537 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:23.309802 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:07:23.309804 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:07:23.309902 (+ 98us) client_negotiation.cc:769] Sending connection context
0507 02:07:23.309975 (+ 73us) client_negotiation.cc:240] Negotiation successful
0507 02:07:23.310062 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":101,"threads_started":1}
I20240507 02:07:23.310720 6484 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.131.125" port: 46259 } has no permanent_uuid. Determining permanent_uuid...
May 07 02:07:23 dist-test-slave-v0bg krb5kdc[5906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.131.125@KRBTEST.COM
May 07 02:07:23 dist-test-slave-v0bg krb5kdc[5906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047643, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.131.125@KRBTEST.COM for oryx/127.31.131.125@KRBTEST.COM
I20240507 02:07:23.317186 6486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:23.310917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.125:46259 (local address 127.0.0.1:42660)
0507 02:07:23.311066 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:23.311075 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:23.311117 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:23.311222 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:23.311223 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:23.311319 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:07:23.311446 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:23.311447 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.312204 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:23.312205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:23.312505 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:23.312506 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.312556 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:23.312871 (+ 315us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:07:23.312874 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:07:23.314059 (+ 1185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:07:23.316103 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:23.316103 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:23.316104 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:23.316271 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:23.316474 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:23.316475 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:23.316476 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:23.316512 (+ 36us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:23.316790 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:07:23.316791 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:07:23.316830 (+ 39us) client_negotiation.cc:769] Sending connection context
0507 02:07:23.316893 (+ 63us) client_negotiation.cc:240] Negotiation successful
0507 02:07:23.316987 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137}
I20240507 02:07:23.317271 6489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:23.310967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42660 (local address 127.31.131.125:46259)
0507 02:07:23.311078 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:23.311079 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:23.311107 (+ 28us) server_negotiation.cc:407] Connection header received
0507 02:07:23.311150 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:23.311151 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:23.311159 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:23.311216 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:23.311506 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.312136 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:23.312569 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.312735 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:23.314140 (+ 1405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:23.314143 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:23.314144 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:07:23.314149 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:23.316026 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:23.316357 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:23.316359 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:23.316359 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:23.316392 (+ 33us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:23.316584 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:23.316584 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:23.316585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:23.316734 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:23.316813 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:23.316935 (+ 122us) server_negotiation.cc:299] Negotiation successful
0507 02:07:23.317109 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":99,"server-negotiator.queue_time_us":92,"thread_start_us":70,"threads_started":1}
I20240507 02:07:23.317320 6488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:23.309208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42646 (local address 127.31.131.125:46259)
0507 02:07:23.309340 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:23.309347 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:23.309362 (+ 15us) server_negotiation.cc:407] Connection header received
0507 02:07:23.309398 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:23.309399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:23.309410 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:23.309463 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:23.310212 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.311003 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:23.311637 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.311772 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:23.313753 (+ 1981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:23.313762 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:23.313762 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:07:23.313769 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:23.316065 (+ 2296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:23.316357 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:23.316359 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:23.316359 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:23.316392 (+ 33us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:23.316584 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:23.316584 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:23.316585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:23.316727 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:23.316851 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:23.316935 (+ 84us) server_negotiation.cc:299] Negotiation successful
0507 02:07:23.317046 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":105,"thread_start_us":79,"threads_started":1}
I20240507 02:07:23.318413 6484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec: Bootstrap starting.
I20240507 02:07:23.318931 6484 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec: Neither blocks nor log segments found. Creating new log.
I20240507 02:07:23.319159 6484 log.cc:824] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec: Log is configured to *not* fsync() on all Append() calls
I20240507 02:07:23.319747 6484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec: No bootstrap required, opened a new log
I20240507 02:07:23.320931 6484 raft_consensus.cc:348] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42c4002f2beb495880b7c5db3cabab77" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 40295 } } peers { permanent_uuid: "063c163f1a014ed18bce90c55921d3ec" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 46259 } }
I20240507 02:07:23.321035 6484 raft_consensus.cc:374] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:07:23.321055 6484 raft_consensus.cc:729] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 063c163f1a014ed18bce90c55921d3ec, State: Initialized, Role: FOLLOWER
I20240507 02:07:23.321152 6484 consensus_queue.cc:260] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42c4002f2beb495880b7c5db3cabab77" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 40295 } } peers { permanent_uuid: "063c163f1a014ed18bce90c55921d3ec" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 46259 } }
I20240507 02:07:23.321390 6484 sys_catalog.cc:564] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec [sys.catalog]: configured and running, proceeding with master startup.
I20240507 02:07:23.321367 6491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42c4002f2beb495880b7c5db3cabab77" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 40295 } } peers { permanent_uuid: "063c163f1a014ed18bce90c55921d3ec" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 46259 } } }
I20240507 02:07:23.321445 6491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec [sys.catalog]: This master's current role is: FOLLOWER
W20240507 02:07:23.322849 6502 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240507 02:07:23.322898 6502 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240507 02:07:23.324147 6416 master_runner.cc:428] Detected that this master 063c163f1a014ed18bce90c55921d3ec is joining an existing cluster
I20240507 02:07:23.324184 6416 master_runner.cc:432] Initiating AddMaster RPC to add 127.31.131.125:46259
I20240507 02:07:23.330214 6509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:23.325182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.125:46259 (local address 127.0.0.1:42668)
0507 02:07:23.325354 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:23.325363 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:23.325415 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:23.325555 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:23.325556 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:23.325736 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:07:23.325989 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:23.325990 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.326817 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:23.326817 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:23.327298 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:23.327299 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.327363 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:23.327759 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:07:23.327763 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:07:23.327970 (+ 207us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:07:23.329528 (+ 1558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:23.329528 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:23.329529 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:23.329646 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:23.329782 (+ 136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:23.329783 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:23.329783 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:23.329802 (+ 19us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:23.330007 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:07:23.330007 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:07:23.330034 (+ 27us) client_negotiation.cc:769] Sending connection context
0507 02:07:23.330087 (+ 53us) client_negotiation.cc:240] Negotiation successful
0507 02:07:23.330150 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":77,"threads_started":1}
I20240507 02:07:23.330320 6489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:23.325339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42668 (local address 127.31.131.125:46259)
0507 02:07:23.325371 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:23.325372 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:23.325421 (+ 49us) server_negotiation.cc:407] Connection header received
0507 02:07:23.325463 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:23.325464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:23.325473 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:23.325537 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:23.326065 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.326743 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:23.327366 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.327543 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:23.328041 (+ 498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:23.328044 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:23.328045 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:07:23.328050 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:23.329451 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:23.329706 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:23.329706 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:23.329707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:23.329729 (+ 22us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:23.329834 (+ 105us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:23.329835 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:23.329835 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:23.329953 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:23.330022 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:23.330114 (+ 92us) server_negotiation.cc:299] Negotiation successful
0507 02:07:23.330215 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":17}
I20240507 02:07:23.357326 5991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:23.325266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49754 (local address 127.31.131.126:40295)
0507 02:07:23.325308 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:23.325309 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:23.325316 (+ 7us) server_negotiation.cc:407] Connection header received
0507 02:07:23.325384 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:23.325385 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:23.325394 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:23.325420 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:23.326085 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.326523 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:23.327161 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.327320 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:23.355289 (+ 27969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:23.355294 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:23.355294 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:07:23.355301 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:23.356476 (+ 1175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:23.356750 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:23.356751 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:23.356751 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:23.356771 (+ 20us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:23.356895 (+ 124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:23.356896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:23.356896 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:23.356992 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:23.357043 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:23.357169 (+ 126us) server_negotiation.cc:299] Negotiation successful
0507 02:07:23.357238 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20240507 02:07:23.357339 6508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:23.325134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.0.0.1:49754)
0507 02:07:23.325255 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:23.325264 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:23.325312 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:23.325494 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:23.325494 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:23.325701 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:07:23.325988 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:23.325990 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.326624 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:23.326625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:23.327074 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:23.327075 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.327361 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:23.354960 (+ 27599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:07:23.354963 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:07:23.355199 (+ 236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:07:23.356549 (+ 1350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:23.356549 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:23.356550 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:23.356693 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:23.356818 (+ 125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:23.356819 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:23.356819 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:23.356846 (+ 27us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:23.357039 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:07:23.357040 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:07:23.357081 (+ 41us) client_negotiation.cc:769] Sending connection context
0507 02:07:23.357139 (+ 58us) client_negotiation.cc:240] Negotiation successful
0507 02:07:23.357231 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":70,"threads_started":1}
May 07 02:07:23 dist-test-slave-v0bg krb5kdc[5906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047642, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.131.126@KRBTEST.COM for oryx/127.31.131.125@KRBTEST.COM
I20240507 02:07:23.365320 6489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:23.359441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42674 (local address 127.31.131.125:46259)
0507 02:07:23.359481 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:23.359482 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:23.359640 (+ 158us) server_negotiation.cc:407] Connection header received
0507 02:07:23.359721 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:23.359722 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:23.359729 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:23.359772 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:23.360486 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.360870 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:23.361528 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.361624 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:23.363352 (+ 1728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:23.363354 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:23.363355 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:07:23.363360 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:23.364474 (+ 1114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:23.364765 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:23.364766 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:23.364767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:23.364785 (+ 18us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:23.364899 (+ 114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:23.364900 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:23.364900 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:23.364982 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:23.365034 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:23.365174 (+ 140us) server_negotiation.cc:299] Negotiation successful
0507 02:07:23.365236 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20240507 02:07:23.365324 6510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:23.359391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.125:46259 (local address 127.0.0.1:42674)
0507 02:07:23.359562 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:23.359584 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:23.359652 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:23.359792 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:23.359795 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:23.360184 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:07:23.360421 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:23.360422 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.360931 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:23.360933 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:23.361465 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:23.361468 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.361538 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:23.361918 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:07:23.361923 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:07:23.363279 (+ 1356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:07:23.364543 (+ 1264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:23.364548 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:23.364551 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:23.364708 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:23.364832 (+ 124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:23.364833 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:23.364834 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:23.364857 (+ 23us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:23.365031 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:07:23.365036 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:07:23.365080 (+ 44us) client_negotiation.cc:769] Sending connection context
0507 02:07:23.365146 (+ 66us) client_negotiation.cc:240] Negotiation successful
0507 02:07:23.365232 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":69,"threads_started":1}
I20240507 02:07:23.365922 5950 catalog_manager.cc:6968] Initiating ChangeConfig request to add master 127.31.131.125:46259: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "063c163f1a014ed18bce90c55921d3ec"
member_type: NON_VOTER
last_known_addr {
host: "127.31.131.125"
port: 46259
}
attrs {
promote: true
}
}
dest_uuid: "42c4002f2beb495880b7c5db3cabab77"
cas_config_opid_index: -1
I20240507 02:07:23.366156 5950 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42c4002f2beb495880b7c5db3cabab77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "42c4002f2beb495880b7c5db3cabab77" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 40295 } } peers { permanent_uuid: "063c163f1a014ed18bce90c55921d3ec" member_type: NON_VOTER last_known_addr { host: "127.31.131.125" port: 46259 } attrs { promote: true } }
I20240507 02:07:23.366569 6511 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 42c4002f2beb495880b7c5db3cabab77 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 063c163f1a014ed18bce90c55921d3ec (127.31.131.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "42c4002f2beb495880b7c5db3cabab77" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 40295 } } peers { permanent_uuid: "063c163f1a014ed18bce90c55921d3ec" member_type: NON_VOTER last_known_addr { host: "127.31.131.125" port: 46259 } attrs { promote: true } } }
I20240507 02:07:23.366850 6458 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:07:23.367537 6511 catalog_manager.cc:6972] Successfully completed master ChangeConfig request to add master 127.31.131.125:46259
I20240507 02:07:23.367636 6458 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec [term 1 FOLLOWER]: Refusing update from remote peer 42c4002f2beb495880b7c5db3cabab77: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240507 02:07:23.367614 6511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42c4002f2beb495880b7c5db3cabab77 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "42c4002f2beb495880b7c5db3cabab77" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "42c4002f2beb495880b7c5db3cabab77" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 40295 } } peers { permanent_uuid: "063c163f1a014ed18bce90c55921d3ec" member_type: NON_VOTER last_known_addr { host: "127.31.131.125" port: 46259 } attrs { promote: true } } }
I20240507 02:07:23.367671 6511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42c4002f2beb495880b7c5db3cabab77 [sys.catalog]: This master's current role is: LEADER
I20240507 02:07:23.367758 6416 master.cc:537] Master@127.31.131.125:46259 shutting down...
I20240507 02:07:23.367884 6512 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 42c4002f2beb495880b7c5db3cabab77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "063c163f1a014ed18bce90c55921d3ec" member_type: NON_VOTER last_known_addr { host: "127.31.131.125" port: 46259 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20240507 02:07:23.368566 5935 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20240507 02:07:23.368911 5935 consensus_peers.cc:485] T 00000000000000000000000000000000 P 42c4002f2beb495880b7c5db3cabab77 -> Peer 063c163f1a014ed18bce90c55921d3ec (127.31.131.125:46259): Couldn't send request to peer 063c163f1a014ed18bce90c55921d3ec. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20240507 02:07:23.370774 6416 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 02:07:23.370846 6416 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 02:07:23.370888 6416 tablet_replica.cc:330] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec: stopping tablet replica
W20240507 02:07:23.372186 6416 connection.cc:441] server connection from 127.0.0.1:42646 torn down before Call kudu.master.MasterService.ListTables from 127.0.0.1:42646 (request call id 23) could send its response
I20240507 02:07:23.383154 6416 master.cc:552] Master@127.31.131.125:46259 shutdown complete.
I20240507 02:07:23.383198 6416 master_runner.cc:305] Clearing existing system tablet
I20240507 02:07:23.383323 6416 fs_manager.cc:359] Metadata directory not provided
I20240507 02:07:23.383368 6416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:07:23.383996 6416 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20240507 02:07:23.384438 6516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:23.384553 6416 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:07:23.384606 6416 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:23.384627 6416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/master-1/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/master-1/wal
uuid: "063c163f1a014ed18bce90c55921d3ec"
format_stamp: "Formatted at 2024-05-07 02:07:23 on dist-test-slave-v0bg"
server_key: "799338019017974c19185ad6bc6e79b0"
server_key_iv: "3cd8d8bea26b09988b968aca56f239b1"
server_key_version: "encryptionkey@0"
I20240507 02:07:23.385299 6416 ts_tablet_manager.cc:1875] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 02:07:23.386677 6416 ts_tablet_manager.cc:1888] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20240507 02:07:23.386740 6416 log.cc:1196] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec: Deleting WAL directory at /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20240507 02:07:23.386919 6416 ts_tablet_manager.cc:1909] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec: Deleting consensus metadata
I20240507 02:07:23.387024 6416 master_runner.cc:315] Copying system tablet from 127.31.131.126:40295
I20240507 02:07:23.387602 6416 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec: tablet copy: Beginning tablet copy session from remote peer at address 127.31.131.126:40295
I20240507 02:07:23.392165 5991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:23.387904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49760 (local address 127.31.131.126:40295)
0507 02:07:23.387946 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:23.387948 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:23.387988 (+ 40us) server_negotiation.cc:407] Connection header received
0507 02:07:23.388034 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:23.388035 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:23.388044 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:23.388106 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:23.388471 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.388927 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:23.389421 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.389571 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:23.390172 (+ 601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:23.390176 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:23.390176 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:07:23.390182 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:23.391351 (+ 1169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:23.391608 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:23.391609 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:23.391609 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:23.391628 (+ 19us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:23.391764 (+ 136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:23.391765 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:23.391765 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:23.391842 (+ 77us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:23.391892 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:23.392011 (+ 119us) server_negotiation.cc:299] Negotiation successful
0507 02:07:23.392076 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20240507 02:07:23.392179 6522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:23.387825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.0.0.1:49760)
0507 02:07:23.387933 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:23.387942 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:23.387996 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:23.388113 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:23.388114 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:23.388235 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:07:23.388403 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:23.388405 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.388998 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:23.388999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:23.389359 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:23.389360 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.389414 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:23.389875 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:07:23.389878 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:07:23.390098 (+ 220us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:07:23.391419 (+ 1321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:23.391420 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:23.391421 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:23.391554 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:23.391677 (+ 123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:23.391678 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:23.391678 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:23.391718 (+ 40us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:23.391889 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:07:23.391890 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:07:23.391927 (+ 37us) client_negotiation.cc:769] Sending connection context
0507 02:07:23.391986 (+ 59us) client_negotiation.cc:240] Negotiation successful
0507 02:07:23.392077 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":64,"threads_started":1}
I20240507 02:07:23.392473 5971 tablet_copy_service.cc:140] P 42c4002f2beb495880b7c5db3cabab77: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 063c163f1a014ed18bce90c55921d3ec ({username='oryx', principal='oryx/127.31.131.125@KRBTEST.COM'} at 127.0.0.1:49760)
I20240507 02:07:23.392532 5971 tablet_copy_service.cc:161] P 42c4002f2beb495880b7c5db3cabab77: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 063c163f1a014ed18bce90c55921d3ec at {username='oryx', principal='oryx/127.31.131.125@KRBTEST.COM'} at 127.0.0.1:49760: session id = 063c163f1a014ed18bce90c55921d3ec-00000000000000000000000000000000
I20240507 02:07:23.393036 5971 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 42c4002f2beb495880b7c5db3cabab77: Tablet Copy: opened 0 blocks and 1 log segments
I20240507 02:07:23.393474 6416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20240507 02:07:23.394583 6416 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec: tablet copy: Starting download of 0 data blocks...
I20240507 02:07:23.394701 6416 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec: tablet copy: Starting download of 1 WAL segments...
I20240507 02:07:23.395432 6416 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240507 02:07:23.396301 5971 tablet_copy_service.cc:342] P 42c4002f2beb495880b7c5db3cabab77: Request end of tablet copy session 063c163f1a014ed18bce90c55921d3ec-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.31.131.125@KRBTEST.COM'} at 127.0.0.1:49760
I20240507 02:07:23.396346 5971 tablet_copy_service.cc:433] P 42c4002f2beb495880b7c5db3cabab77: ending tablet copy session 063c163f1a014ed18bce90c55921d3ec-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 063c163f1a014ed18bce90c55921d3ec
I20240507 02:07:23.398408 6416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:07:23.399377 6524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:07:23.399403 6526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:07:23.399492 6523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:07:23.399729 6416 server_base.cc:1034] running on GCE node
I20240507 02:07:23.399825 6416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:07:23.399935 6416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:07:23.401058 6416 hybrid_clock.cc:648] HybridClock initialized: now 1715047643401048 us; error 26 us; skew 500 ppm
May 07 02:07:23 dist-test-slave-v0bg krb5kdc[5906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047643, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.131.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:07:23.402102 6416 init.cc:377] Logged in from keytab as oryx/127.31.131.125@KRBTEST.COM (short username oryx)
I20240507 02:07:23.402302 6416 webserver.cc:460] Webserver started at http://127.31.131.125:39305/ using document root <none> and password file <none>
I20240507 02:07:23.402402 6416 fs_manager.cc:359] Metadata directory not provided
I20240507 02:07:23.402436 6416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:07:23.402987 6416 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:23.403398 6533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:23.403524 6416 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:23.403566 6416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/master-1/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/master-1/wal
uuid: "063c163f1a014ed18bce90c55921d3ec"
format_stamp: "Formatted at 2024-05-07 02:07:23 on dist-test-slave-v0bg"
server_key: "799338019017974c19185ad6bc6e79b0"
server_key_iv: "3cd8d8bea26b09988b968aca56f239b1"
server_key_version: "encryptionkey@0"
I20240507 02:07:23.403611 6416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:07:23.410041 6416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:07:23.411418 6416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:07:23.413904 6416 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.125:46259
I20240507 02:07:23.413924 6585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.125:46259 every 8 connection(s)
I20240507 02:07:23.414170 6416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/master-1/data/info.pb
I20240507 02:07:23.414925 6586 sys_catalog.cc:263] Verifying existing consensus state
I20240507 02:07:23.415380 6586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec: Bootstrap starting.
I20240507 02:07:23.417827 6586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240507 02:07:23.418037 6586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec: Bootstrap complete.
I20240507 02:07:23.418228 6586 raft_consensus.cc:348] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "42c4002f2beb495880b7c5db3cabab77" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 40295 } } peers { permanent_uuid: "063c163f1a014ed18bce90c55921d3ec" member_type: NON_VOTER last_known_addr { host: "127.31.131.125" port: 46259 } attrs { promote: true } }
I20240507 02:07:23.418313 6586 raft_consensus.cc:729] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 063c163f1a014ed18bce90c55921d3ec, State: Initialized, Role: LEARNER
I20240507 02:07:23.418357 6586 consensus_queue.cc:260] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "42c4002f2beb495880b7c5db3cabab77" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 40295 } } peers { permanent_uuid: "063c163f1a014ed18bce90c55921d3ec" member_type: NON_VOTER last_known_addr { host: "127.31.131.125" port: 46259 } attrs { promote: true } }
I20240507 02:07:23.418519 6586 sys_catalog.cc:564] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec [sys.catalog]: configured and running, proceeding with master startup.
I20240507 02:07:23.418515 6589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "42c4002f2beb495880b7c5db3cabab77" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 40295 } } peers { permanent_uuid: "063c163f1a014ed18bce90c55921d3ec" member_type: NON_VOTER last_known_addr { host: "127.31.131.125" port: 46259 } attrs { promote: true } } }
I20240507 02:07:23.418680 6589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec [sys.catalog]: This master's current role is: LEARNER
I20240507 02:07:23.419955 6600 catalog_manager.cc:1259] Loaded cluster ID: 4b358b1693354b3488e6edbc34acd7c3
I20240507 02:07:23.420104 6600 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec: loading cluster ID for follower catalog manager: success
I20240507 02:07:23.421079 6600 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec: acquiring CA information for follower catalog manager: success
I20240507 02:07:23.421312 6600 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240507 02:07:23.427778 6601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:23.422467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42688 (local address 127.31.131.125:46259)
0507 02:07:23.422607 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:23.422608 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:23.422617 (+ 9us) server_negotiation.cc:407] Connection header received
0507 02:07:23.422632 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:23.422633 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:23.422640 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:23.422701 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:23.423244 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.423870 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:23.424351 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.424507 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:23.425347 (+ 840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:23.425350 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:23.425350 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:07:23.425355 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:23.426988 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:23.427240 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:23.427241 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:23.427242 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:23.427262 (+ 20us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:23.427376 (+ 114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:23.427377 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:23.427377 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:23.427452 (+ 75us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:23.427534 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:23.427616 (+ 82us) server_negotiation.cc:299] Negotiation successful
0507 02:07:23.427687 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":67,"threads_started":1}
WARNING: no policy specified for oryx/127.31.131.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.131.124@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.131.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/krb5kdc/oryx_127.31.131.124.keytab.
Entry for principal oryx/127.31.131.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/krb5kdc/oryx_127.31.131.124.keytab.
I20240507 02:07:23.449045 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.124:42683
--webserver_interface=127.31.131.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/krb5kdc/oryx_127.31.131.124.keytab
--principal=oryx/127.31.131.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:42939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.131.126:40295,127.31.131.125:46259,127.31.131.124:42683
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:07:23.523653 6604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:07:23.523816 6604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:07:23.523835 6604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:07:23.525111 6604 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240507 02:07:23.525153 6604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:07:23.525166 6604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:07:23.525177 6604 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240507 02:07:23.525192 6604 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240507 02:07:23.526647 6604 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:42939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.31.131.126:40295,127.31.131.125:46259,127.31.131.124:42683
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/krb5kdc/oryx_127.31.131.124.keytab
--principal=oryx/127.31.131.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.124:42683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.31.131.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:07:23.526831 6604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:07:23.526993 6604 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:07:23.530189 6610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:07:23.530205 6612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:07:23.530359 6604 server_base.cc:1034] running on GCE node
W20240507 02:07:23.530431 6609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:07:23.530630 6604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:07:23.530838 6604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:07:23.531982 6604 hybrid_clock.cc:648] HybridClock initialized: now 1715047643531965 us; error 26 us; skew 500 ppm
May 07 02:07:23 dist-test-slave-v0bg krb5kdc[5906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047643, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.131.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:07:23.534390 6604 init.cc:377] Logged in from keytab as oryx/127.31.131.124@KRBTEST.COM (short username oryx)
I20240507 02:07:23.534808 6604 webserver.cc:460] Webserver started at http://127.31.131.124:35807/ using document root <none> and password file <none>
I20240507 02:07:23.534992 6604 fs_manager.cc:359] Metadata directory not provided
I20240507 02:07:23.535043 6604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:07:23.535144 6604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:07:23.536227 6604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/master-2/data/instance:
uuid: "65ae2553e5234a48bb69f75f1afe1ea5"
format_stamp: "Formatted at 2024-05-07 02:07:23 on dist-test-slave-v0bg"
server_key: "c9aab099fe267ddf76e6f5ea27842e87"
server_key_iv: "348fec3514c65d75987bee9f44f071ee"
server_key_version: "encryptionkey@0"
I20240507 02:07:23.536523 6604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/master-2/wal/instance:
uuid: "65ae2553e5234a48bb69f75f1afe1ea5"
format_stamp: "Formatted at 2024-05-07 02:07:23 on dist-test-slave-v0bg"
server_key: "c9aab099fe267ddf76e6f5ea27842e87"
server_key_iv: "348fec3514c65d75987bee9f44f071ee"
server_key_version: "encryptionkey@0"
I20240507 02:07:23.537974 6604 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240507 02:07:23.538717 6619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:23.538853 6604 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240507 02:07:23.538931 6604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/master-2/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/master-2/wal
uuid: "65ae2553e5234a48bb69f75f1afe1ea5"
format_stamp: "Formatted at 2024-05-07 02:07:23 on dist-test-slave-v0bg"
server_key: "c9aab099fe267ddf76e6f5ea27842e87"
server_key_iv: "348fec3514c65d75987bee9f44f071ee"
server_key_version: "encryptionkey@0"
I20240507 02:07:23.538976 6604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:07:23.547108 6604 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:07:23.548501 6604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:07:23.548585 6604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:07:23.551821 6604 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.124:42683
I20240507 02:07:23.551858 6671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.124:42683 every 8 connection(s)
I20240507 02:07:23.552098 6604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/master-2/data/info.pb
I20240507 02:07:23.552289 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 6604
I20240507 02:07:23.553115 6672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:07:23.554669 6672 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.131.126" port: 40295 } has no permanent_uuid. Determining permanent_uuid...
May 07 02:07:23 dist-test-slave-v0bg krb5kdc[5906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.131.124@KRBTEST.COM
May 07 02:07:23 dist-test-slave-v0bg krb5kdc[5906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047643, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.131.124@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM
I20240507 02:07:23.561228 5991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:23.555365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49776 (local address 127.31.131.126:40295)
0507 02:07:23.555421 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:23.555423 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:23.555434 (+ 11us) server_negotiation.cc:407] Connection header received
0507 02:07:23.555498 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:23.555498 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:23.555508 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:23.555565 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:23.556128 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.556639 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:23.557324 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.557503 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:23.558991 (+ 1488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:23.558995 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:23.558996 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:07:23.559001 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:23.560230 (+ 1229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:23.560556 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:23.560557 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:23.560557 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:23.560577 (+ 20us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:23.560735 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:23.560736 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:23.560736 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:23.560829 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:23.560905 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:23.561064 (+ 159us) server_negotiation.cc:299] Negotiation successful
0507 02:07:23.561144 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20240507 02:07:23.561342 6675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:23.555189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.0.0.1:49776)
0507 02:07:23.555373 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:23.555397 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:23.555445 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:23.555577 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:23.555579 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:23.555811 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:07:23.556042 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:23.556044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.556718 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:23.556719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:23.557246 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:23.557248 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.557317 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:23.557679 (+ 362us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:07:23.557692 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:07:23.558914 (+ 1222us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:07:23.560292 (+ 1378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:23.560297 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:23.560300 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:23.560500 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:23.560628 (+ 128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:23.560629 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:23.560629 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:23.560684 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:23.560886 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:07:23.560889 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:07:23.560945 (+ 56us) client_negotiation.cc:769] Sending connection context
0507 02:07:23.561037 (+ 92us) client_negotiation.cc:240] Negotiation successful
0507 02:07:23.561165 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":69,"threads_started":1}
I20240507 02:07:23.561478 6674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:23.552802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33552 (local address 127.31.131.124:42683)
0507 02:07:23.552943 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:23.552954 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:23.552974 (+ 20us) server_negotiation.cc:407] Connection header received
0507 02:07:23.553465 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:23.553473 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:23.553523 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:23.553683 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:23.554268 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.555324 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:23.555930 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.556136 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:23.558633 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:23.558640 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:23.558640 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:07:23.558645 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:23.560200 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:23.560515 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:23.560518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:23.560518 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:23.560558 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:23.560696 (+ 138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:23.560697 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:23.560697 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:23.560882 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:23.560983 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:23.561083 (+ 100us) server_negotiation.cc:299] Negotiation successful
0507 02:07:23.561235 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":72,"server-negotiator.queue_time_us":103,"thread_start_us":57,"threads_started":1}
I20240507 02:07:23.561784 6672 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.131.125" port: 46259 } has no permanent_uuid. Determining permanent_uuid...
May 07 02:07:23 dist-test-slave-v0bg krb5kdc[5906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047643, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.131.124@KRBTEST.COM for oryx/127.31.131.125@KRBTEST.COM
I20240507 02:07:23.566404 6675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:23.561988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.125:46259 (local address 127.0.0.1:42700)
0507 02:07:23.562018 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:23.562027 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:23.562063 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:23.562212 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:23.562213 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:23.562327 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:07:23.562440 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:23.562441 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.563085 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:23.563086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:23.563401 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:23.563402 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.563461 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:23.563777 (+ 316us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:07:23.563780 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:07:23.564725 (+ 945us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:07:23.565851 (+ 1126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:23.565852 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:23.565852 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:23.565963 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:23.566067 (+ 104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:23.566067 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:23.566067 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:23.566085 (+ 18us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:23.566217 (+ 132us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:07:23.566218 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:07:23.566245 (+ 27us) client_negotiation.cc:769] Sending connection context
0507 02:07:23.566285 (+ 40us) client_negotiation.cc:240] Negotiation successful
0507 02:07:23.566343 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14}
I20240507 02:07:23.566416 6601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:23.562069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42700 (local address 127.31.131.125:46259)
0507 02:07:23.562120 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:23.562121 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:23.562128 (+ 7us) server_negotiation.cc:407] Connection header received
0507 02:07:23.562148 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:23.562149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:23.562164 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:23.562206 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:23.562509 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.563021 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:23.563463 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.563623 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:23.564793 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:23.564795 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:23.564796 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:07:23.564800 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:23.565791 (+ 991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:23.566012 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:23.566012 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:23.566012 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:23.566027 (+ 15us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:23.566114 (+ 87us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:23.566114 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:23.566114 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:23.566181 (+ 67us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:23.566219 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:23.566300 (+ 81us) server_negotiation.cc:299] Negotiation successful
0507 02:07:23.566357 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20240507 02:07:23.566751 6672 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.131.124" port: 42683 } has no permanent_uuid. Determining permanent_uuid...
May 07 02:07:23 dist-test-slave-v0bg krb5kdc[5906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047643, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.131.124@KRBTEST.COM for oryx/127.31.131.124@KRBTEST.COM
I20240507 02:07:23.571779 6675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:23.566953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.124:42683 (local address 127.0.0.1:33562)
0507 02:07:23.566994 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:23.567002 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:23.567035 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:23.567141 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:23.567142 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:23.567251 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:07:23.567348 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:23.567350 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.567866 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:23.567867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:23.568163 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:23.568164 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.568217 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:23.568521 (+ 304us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:07:23.568525 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:07:23.569463 (+ 938us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:07:23.571098 (+ 1635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:23.571099 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:23.571099 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:23.571214 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:23.571327 (+ 113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:23.571328 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:23.571328 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:23.571351 (+ 23us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:23.571517 (+ 166us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:07:23.571517 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:07:23.571548 (+ 31us) client_negotiation.cc:769] Sending connection context
0507 02:07:23.571606 (+ 58us) client_negotiation.cc:240] Negotiation successful
0507 02:07:23.571689 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20240507 02:07:23.571831 6674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:23.567006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33562 (local address 127.31.131.124:42683)
0507 02:07:23.567033 (+ 27us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:23.567034 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:23.567041 (+ 7us) server_negotiation.cc:407] Connection header received
0507 02:07:23.567084 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:23.567085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:23.567092 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:23.567139 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:23.567408 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.567811 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:23.568221 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.568366 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:23.569531 (+ 1165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:23.569534 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:23.569534 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:07:23.569539 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:23.571031 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:23.571266 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:23.571266 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:23.571267 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:23.571283 (+ 16us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:23.571405 (+ 122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:23.571406 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:23.571406 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:23.571472 (+ 66us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:23.571517 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:23.571629 (+ 112us) server_negotiation.cc:299] Negotiation successful
0507 02:07:23.571732 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":15}
I20240507 02:07:23.572871 6672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65ae2553e5234a48bb69f75f1afe1ea5: Bootstrap starting.
I20240507 02:07:23.573336 6672 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65ae2553e5234a48bb69f75f1afe1ea5: Neither blocks nor log segments found. Creating new log.
I20240507 02:07:23.573560 6672 log.cc:824] T 00000000000000000000000000000000 P 65ae2553e5234a48bb69f75f1afe1ea5: Log is configured to *not* fsync() on all Append() calls
I20240507 02:07:23.574168 6672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65ae2553e5234a48bb69f75f1afe1ea5: No bootstrap required, opened a new log
I20240507 02:07:23.575409 6672 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65ae2553e5234a48bb69f75f1afe1ea5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42c4002f2beb495880b7c5db3cabab77" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 40295 } } peers { permanent_uuid: "063c163f1a014ed18bce90c55921d3ec" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 46259 } } peers { permanent_uuid: "65ae2553e5234a48bb69f75f1afe1ea5" member_type: VOTER last_known_addr { host: "127.31.131.124" port: 42683 } }
I20240507 02:07:23.575507 6672 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65ae2553e5234a48bb69f75f1afe1ea5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:07:23.575527 6672 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65ae2553e5234a48bb69f75f1afe1ea5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65ae2553e5234a48bb69f75f1afe1ea5, State: Initialized, Role: FOLLOWER
I20240507 02:07:23.575608 6672 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65ae2553e5234a48bb69f75f1afe1ea5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42c4002f2beb495880b7c5db3cabab77" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 40295 } } peers { permanent_uuid: "063c163f1a014ed18bce90c55921d3ec" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 46259 } } peers { permanent_uuid: "65ae2553e5234a48bb69f75f1afe1ea5" member_type: VOTER last_known_addr { host: "127.31.131.124" port: 42683 } }
I20240507 02:07:23.575835 6672 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65ae2553e5234a48bb69f75f1afe1ea5 [sys.catalog]: configured and running, proceeding with master startup.
I20240507 02:07:23.575803 6677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65ae2553e5234a48bb69f75f1afe1ea5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42c4002f2beb495880b7c5db3cabab77" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 40295 } } peers { permanent_uuid: "063c163f1a014ed18bce90c55921d3ec" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 46259 } } peers { permanent_uuid: "65ae2553e5234a48bb69f75f1afe1ea5" member_type: VOTER last_known_addr { host: "127.31.131.124" port: 42683 } } }
I20240507 02:07:23.575891 6677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65ae2553e5234a48bb69f75f1afe1ea5 [sys.catalog]: This master's current role is: FOLLOWER
W20240507 02:07:23.577427 6688 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 65ae2553e5234a48bb69f75f1afe1ea5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240507 02:07:23.577479 6688 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240507 02:07:23.579277 6604 master_runner.cc:428] Detected that this master 65ae2553e5234a48bb69f75f1afe1ea5 is joining an existing cluster
I20240507 02:07:23.579321 6604 master_runner.cc:432] Initiating AddMaster RPC to add 127.31.131.124:42683
I20240507 02:07:23.612661 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 6013
I20240507 02:07:23.617183 6674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:23.580343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33568 (local address 127.31.131.124:42683)
0507 02:07:23.580378 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:23.580380 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:23.580736 (+ 356us) server_negotiation.cc:407] Connection header received
0507 02:07:23.580802 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:23.580803 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:23.580809 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:23.580830 (+ 21us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:23.611481 (+ 30651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.612091 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:23.612888 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.613040 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:23.614380 (+ 1340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:23.614383 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:23.614383 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:07:23.614389 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:23.616325 (+ 1936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:23.616636 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:23.616637 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:23.616637 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:23.616654 (+ 17us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:23.616765 (+ 111us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:23.616766 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:23.616766 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:23.616844 (+ 78us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:23.616874 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:23.617027 (+ 153us) server_negotiation.cc:299] Negotiation successful
0507 02:07:23.617092 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20240507 02:07:23.617234 6601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:23.580472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42710 (local address 127.31.131.125:46259)
0507 02:07:23.580510 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:23.580512 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:23.580601 (+ 89us) server_negotiation.cc:407] Connection header received
0507 02:07:23.580661 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:23.580661 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:23.580671 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:23.580718 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:23.611324 (+ 30606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.612100 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:23.613184 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.613365 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:23.614401 (+ 1036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:23.614405 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:23.614406 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:07:23.614412 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:23.615920 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:23.616546 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:23.616548 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:23.616548 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:23.616571 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:23.616724 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:23.616725 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:23.616725 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:23.616824 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:23.616861 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:23.617051 (+ 190us) server_negotiation.cc:299] Negotiation successful
0507 02:07:23.617136 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20240507 02:07:23.617504 6694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:23.580278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.125:46259 (local address 127.0.0.1:42710)
0507 02:07:23.580534 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:23.580542 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:23.580607 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:23.580761 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:23.580762 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:23.610891 (+ 30129us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:07:23.611186 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:23.611190 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.612198 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:23.612199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:23.613083 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:23.613086 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.613423 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:23.613805 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:07:23.613809 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:07:23.614298 (+ 489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:07:23.616005 (+ 1707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:23.616006 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:23.616006 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:23.616477 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:23.616638 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:23.616639 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:23.616639 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:23.616667 (+ 28us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:23.616899 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:07:23.616900 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:07:23.616939 (+ 39us) client_negotiation.cc:769] Sending connection context
0507 02:07:23.617305 (+ 366us) client_negotiation.cc:240] Negotiation successful
0507 02:07:23.617404 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"spinlock_wait_cycles":7040,"thread_start_us":90,"threads_started":1}
I20240507 02:07:23.617694 6695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:23.580405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.124:42683 (local address 127.0.0.1:33568)
0507 02:07:23.580688 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:23.580695 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:23.580752 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:23.580875 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:23.580876 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:23.610894 (+ 30018us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:07:23.611406 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:23.611408 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.612167 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:23.612167 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:23.612813 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:23.612816 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.613081 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:23.613823 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:07:23.613827 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:07:23.614300 (+ 473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:07:23.616389 (+ 2089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:23.616390 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:23.616390 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:23.616561 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:23.616701 (+ 140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:23.616702 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:23.616702 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:23.616722 (+ 20us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:23.616904 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:07:23.616904 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:07:23.616932 (+ 28us) client_negotiation.cc:769] Sending connection context
0507 02:07:23.617438 (+ 506us) client_negotiation.cc:240] Negotiation successful
0507 02:07:23.617522 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":74,"threads_started":1}
I20240507 02:07:23.620291 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.65:38131
--local_ip_for_outbound_sockets=127.31.131.65
--tserver_master_addrs=127.31.131.126:40295
--webserver_port=45227
--webserver_interface=127.31.131.65
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/krb5kdc/oryx_127.31.131.65.keytab
--principal=oryx/127.31.131.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:42939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.31.131.126:40295,127.31.131.125:46259
--tserver_master_addrs=127.31.131.126:40295,127.31.131.125:46259,127.31.131.124:42683 with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240507 02:07:23.622713 6696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:23.580278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.0.0.1:49782)
0507 02:07:23.580534 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:23.580543 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:23.580626 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:23.580792 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:23.580793 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:23.610900 (+ 30107us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:07:23.611170 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:23.611176 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.612241 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:23.612241 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:23.619332 (+ 7091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:23.619335 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.619419 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:23.619847 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:07:23.619851 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:07:23.620180 (+ 329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:07:23.621847 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:23.621848 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:23.621848 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:23.622018 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:23.622195 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:23.622195 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:23.622196 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:23.622223 (+ 27us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:23.622420 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:07:23.622421 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:07:23.622460 (+ 39us) client_negotiation.cc:769] Sending connection context
0507 02:07:23.622521 (+ 61us) client_negotiation.cc:240] Negotiation successful
0507 02:07:23.622625 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":2,"thread_start_us":86,"threads_started":1}
I20240507 02:07:23.622740 5991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:23.580465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49782 (local address 127.31.131.126:40295)
0507 02:07:23.580520 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:23.580522 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:23.580602 (+ 80us) server_negotiation.cc:407] Connection header received
0507 02:07:23.580693 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:23.580694 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:23.580704 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:23.580730 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:23.611330 (+ 30600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.612097 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:23.619427 (+ 7330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.619920 (+ 493us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:23.620275 (+ 355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:23.620280 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:23.620280 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:07:23.620287 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:23.621767 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:23.622109 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:23.622110 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:23.622110 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:23.622137 (+ 27us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:23.622265 (+ 128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:23.622265 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:23.622266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:23.622364 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:23.622427 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:23.622570 (+ 143us) server_negotiation.cc:299] Negotiation successful
0507 02:07:23.622648 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
May 07 02:07:23 dist-test-slave-v0bg krb5kdc[5906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047642, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.131.126@KRBTEST.COM for oryx/127.31.131.124@KRBTEST.COM
I20240507 02:07:23.631006 6510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:23.624644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.124:42683 (local address 127.0.0.1:33572)
0507 02:07:23.624688 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:23.624697 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:23.624731 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:23.624877 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:23.624877 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:23.625031 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:07:23.625219 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:23.625221 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.625991 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:23.625991 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:23.626521 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:23.626523 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.626686 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:23.627076 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:07:23.627081 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:07:23.628489 (+ 1408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:07:23.630131 (+ 1642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:23.630132 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:23.630132 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:23.630293 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:23.630442 (+ 149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:23.630442 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:23.630443 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:23.630472 (+ 29us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:23.630718 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:07:23.630719 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:07:23.630758 (+ 39us) client_negotiation.cc:769] Sending connection context
0507 02:07:23.630814 (+ 56us) client_negotiation.cc:240] Negotiation successful
0507 02:07:23.630900 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20240507 02:07:23.631263 6674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:23.624695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33572 (local address 127.31.131.124:42683)
0507 02:07:23.624728 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:23.624730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:23.624739 (+ 9us) server_negotiation.cc:407] Connection header received
0507 02:07:23.624798 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:23.624799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:23.624808 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:23.624868 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:23.625289 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.625910 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:23.626695 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.627147 (+ 452us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:23.628575 (+ 1428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:23.628578 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:23.628579 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:07:23.628584 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:23.630046 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:23.630363 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:23.630364 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:23.630364 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:23.630385 (+ 21us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:23.630528 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:23.630529 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:23.630529 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:23.630640 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:23.630694 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:23.631119 (+ 425us) server_negotiation.cc:299] Negotiation successful
0507 02:07:23.631186 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20240507 02:07:23.631755 5950 catalog_manager.cc:6968] Initiating ChangeConfig request to add master 127.31.131.124:42683: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "65ae2553e5234a48bb69f75f1afe1ea5"
member_type: NON_VOTER
last_known_addr {
host: "127.31.131.124"
port: 42683
}
attrs {
promote: true
}
}
dest_uuid: "42c4002f2beb495880b7c5db3cabab77"
cas_config_opid_index: 5
I20240507 02:07:23.631996 5950 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42c4002f2beb495880b7c5db3cabab77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "42c4002f2beb495880b7c5db3cabab77" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 40295 } } peers { permanent_uuid: "063c163f1a014ed18bce90c55921d3ec" member_type: NON_VOTER last_known_addr { host: "127.31.131.125" port: 46259 } attrs { promote: true } } peers { permanent_uuid: "65ae2553e5234a48bb69f75f1afe1ea5" member_type: NON_VOTER last_known_addr { host: "127.31.131.124" port: 42683 } attrs { promote: true } }
I20240507 02:07:23.632449 6512 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 42c4002f2beb495880b7c5db3cabab77 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 65ae2553e5234a48bb69f75f1afe1ea5 (127.31.131.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "42c4002f2beb495880b7c5db3cabab77" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 40295 } } peers { permanent_uuid: "063c163f1a014ed18bce90c55921d3ec" member_type: NON_VOTER last_known_addr { host: "127.31.131.125" port: 46259 } attrs { promote: true } } peers { permanent_uuid: "65ae2553e5234a48bb69f75f1afe1ea5" member_type: NON_VOTER last_known_addr { host: "127.31.131.124" port: 42683 } attrs { promote: true } } }
I20240507 02:07:23.633405 6512 catalog_manager.cc:6972] Successfully completed master ChangeConfig request to add master 127.31.131.124:42683
I20240507 02:07:23.633502 6698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42c4002f2beb495880b7c5db3cabab77 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "42c4002f2beb495880b7c5db3cabab77" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "42c4002f2beb495880b7c5db3cabab77" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 40295 } } peers { permanent_uuid: "063c163f1a014ed18bce90c55921d3ec" member_type: NON_VOTER last_known_addr { host: "127.31.131.125" port: 46259 } attrs { promote: true } } peers { permanent_uuid: "65ae2553e5234a48bb69f75f1afe1ea5" member_type: NON_VOTER last_known_addr { host: "127.31.131.124" port: 42683 } attrs { promote: true } } }
I20240507 02:07:23.633657 6698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42c4002f2beb495880b7c5db3cabab77 [sys.catalog]: This master's current role is: LEADER
I20240507 02:07:23.633865 6604 master.cc:537] Master@127.31.131.124:42683 shutting down...
I20240507 02:07:23.633963 6646 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65ae2553e5234a48bb69f75f1afe1ea5 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:07:23.634944 6646 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 65ae2553e5234a48bb69f75f1afe1ea5 [term 1 FOLLOWER]: Refusing update from remote peer 42c4002f2beb495880b7c5db3cabab77: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240507 02:07:23.635360 6698 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 42c4002f2beb495880b7c5db3cabab77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65ae2553e5234a48bb69f75f1afe1ea5" member_type: NON_VOTER last_known_addr { host: "127.31.131.124" port: 42683 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20240507 02:07:23.636083 5938 consensus_peers.cc:485] T 00000000000000000000000000000000 P 42c4002f2beb495880b7c5db3cabab77 -> Peer 65ae2553e5234a48bb69f75f1afe1ea5 (127.31.131.124:42683): Couldn't send request to peer 65ae2553e5234a48bb69f75f1afe1ea5. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20240507 02:07:23.637076 6604 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 65ae2553e5234a48bb69f75f1afe1ea5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 02:07:23.637151 6604 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 65ae2553e5234a48bb69f75f1afe1ea5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 02:07:23.637187 6604 tablet_replica.cc:330] T 00000000000000000000000000000000 P 65ae2553e5234a48bb69f75f1afe1ea5: stopping tablet replica
I20240507 02:07:23.639873 6510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:23.632577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.125:46259 (local address 127.0.0.1:42720)
0507 02:07:23.636882 (+ 4305us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:23.636893 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:23.636940 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:23.637155 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:23.637156 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:23.637157 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:07:23.637322 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:23.637323 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.638209 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:23.638211 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:23.639083 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:23.639085 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.639689 (+ 604us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:23.639729 (+ 40us) client_negotiation.cc:769] Sending connection context
0507 02:07:23.639778 (+ 49us) client_negotiation.cc:240] Negotiation successful
0507 02:07:23.639816 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4286}
I20240507 02:07:23.640064 6601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:23.633568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42720 (local address 127.31.131.125:46259)
0507 02:07:23.633670 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:23.633672 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:23.636995 (+ 3323us) server_negotiation.cc:407] Connection header received
0507 02:07:23.637021 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:23.637022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:23.637048 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:23.637096 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:07:23.637394 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.638108 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:23.639176 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.639590 (+ 414us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:23.639658 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:23.639968 (+ 310us) server_negotiation.cc:299] Negotiation successful
0507 02:07:23.640007 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20240507 02:07:23.640250 6560 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec [term 1 LEARNER]: Refusing update from remote peer 42c4002f2beb495880b7c5db3cabab77: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240507 02:07:23.640458 6698 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 42c4002f2beb495880b7c5db3cabab77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "063c163f1a014ed18bce90c55921d3ec" member_type: NON_VOTER last_known_addr { host: "127.31.131.125" port: 46259 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240507 02:07:23.640973 6560 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 65ae2553e5234a48bb69f75f1afe1ea5 (127.31.131.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "42c4002f2beb495880b7c5db3cabab77" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 40295 } } peers { permanent_uuid: "063c163f1a014ed18bce90c55921d3ec" member_type: NON_VOTER last_known_addr { host: "127.31.131.125" port: 46259 } attrs { promote: true } } peers { permanent_uuid: "65ae2553e5234a48bb69f75f1afe1ea5" member_type: NON_VOTER last_known_addr { host: "127.31.131.124" port: 42683 } attrs { promote: true } } }
I20240507 02:07:23.642630 6700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "42c4002f2beb495880b7c5db3cabab77" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "42c4002f2beb495880b7c5db3cabab77" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 40295 } } peers { permanent_uuid: "063c163f1a014ed18bce90c55921d3ec" member_type: NON_VOTER last_known_addr { host: "127.31.131.125" port: 46259 } attrs { promote: true } } peers { permanent_uuid: "65ae2553e5234a48bb69f75f1afe1ea5" member_type: NON_VOTER last_known_addr { host: "127.31.131.124" port: 42683 } attrs { promote: true } } }
I20240507 02:07:23.642702 6700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec [sys.catalog]: This master's current role is: LEARNER
I20240507 02:07:23.642663 6589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42c4002f2beb495880b7c5db3cabab77. Latest consensus state: current_term: 1 leader_uuid: "42c4002f2beb495880b7c5db3cabab77" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "42c4002f2beb495880b7c5db3cabab77" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 40295 } } peers { permanent_uuid: "063c163f1a014ed18bce90c55921d3ec" member_type: NON_VOTER last_known_addr { host: "127.31.131.125" port: 46259 } attrs { promote: true } } peers { permanent_uuid: "65ae2553e5234a48bb69f75f1afe1ea5" member_type: NON_VOTER last_known_addr { host: "127.31.131.124" port: 42683 } attrs { promote: true } } }
I20240507 02:07:23.642743 6589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec [sys.catalog]: This master's current role is: LEARNER
I20240507 02:07:23.642959 6698 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 42c4002f2beb495880b7c5db3cabab77: attempting to promote NON_VOTER 063c163f1a014ed18bce90c55921d3ec to VOTER
I20240507 02:07:23.643105 6698 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42c4002f2beb495880b7c5db3cabab77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "42c4002f2beb495880b7c5db3cabab77" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 40295 } } peers { permanent_uuid: "063c163f1a014ed18bce90c55921d3ec" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 46259 } attrs { promote: false } } peers { permanent_uuid: "65ae2553e5234a48bb69f75f1afe1ea5" member_type: NON_VOTER last_known_addr { host: "127.31.131.124" port: 42683 } attrs { promote: true } }
I20240507 02:07:23.643455 6510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:23.643344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.124:42683 (local address 127.0.0.1:33588)
0507 02:07:23.643389 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:23.643429 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.124:42683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20240507 02:07:23.643519 6560 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec [term 1 LEARNER]: Refusing update from remote peer 42c4002f2beb495880b7c5db3cabab77: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240507 02:07:23.643666 6512 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 42c4002f2beb495880b7c5db3cabab77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "063c163f1a014ed18bce90c55921d3ec" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 46259 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240507 02:07:23.643842 6510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:23.643738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.124:42683 (local address 127.0.0.1:33590)
0507 02:07:23.643802 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:23.643822 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.124:42683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20240507 02:07:23.643966 5938 consensus_peers.cc:485] T 00000000000000000000000000000000 P 42c4002f2beb495880b7c5db3cabab77 -> Peer 65ae2553e5234a48bb69f75f1afe1ea5 (127.31.131.124:42683): Couldn't send request to peer 65ae2553e5234a48bb69f75f1afe1ea5. Status: Network error: Client connection negotiation failed: client connection to 127.31.131.124:42683: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240507 02:07:23.644272 6512 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 42c4002f2beb495880b7c5db3cabab77 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 063c163f1a014ed18bce90c55921d3ec (127.31.131.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "42c4002f2beb495880b7c5db3cabab77" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 40295 } } peers { permanent_uuid: "063c163f1a014ed18bce90c55921d3ec" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 46259 } attrs { promote: false } } peers { permanent_uuid: "65ae2553e5234a48bb69f75f1afe1ea5" member_type: NON_VOTER last_known_addr { host: "127.31.131.124" port: 42683 } attrs { promote: true } } }
I20240507 02:07:23.645040 6512 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42c4002f2beb495880b7c5db3cabab77 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "42c4002f2beb495880b7c5db3cabab77" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "42c4002f2beb495880b7c5db3cabab77" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 40295 } } peers { permanent_uuid: "063c163f1a014ed18bce90c55921d3ec" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 46259 } attrs { promote: false } } peers { permanent_uuid: "65ae2553e5234a48bb69f75f1afe1ea5" member_type: NON_VOTER last_known_addr { host: "127.31.131.124" port: 42683 } attrs { promote: true } } }
I20240507 02:07:23.645102 6512 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42c4002f2beb495880b7c5db3cabab77 [sys.catalog]: This master's current role is: LEADER
I20240507 02:07:23.645150 6512 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42c4002f2beb495880b7c5db3cabab77 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "42c4002f2beb495880b7c5db3cabab77" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "42c4002f2beb495880b7c5db3cabab77" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 40295 } } peers { permanent_uuid: "063c163f1a014ed18bce90c55921d3ec" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 46259 } attrs { promote: false } } peers { permanent_uuid: "65ae2553e5234a48bb69f75f1afe1ea5" member_type: NON_VOTER last_known_addr { host: "127.31.131.124" port: 42683 } attrs { promote: true } } }
I20240507 02:07:23.645184 6512 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42c4002f2beb495880b7c5db3cabab77 [sys.catalog]: This master's current role is: LEADER
I20240507 02:07:23.645455 6560 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 063c163f1a014ed18bce90c55921d3ec (127.31.131.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "42c4002f2beb495880b7c5db3cabab77" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 40295 } } peers { permanent_uuid: "063c163f1a014ed18bce90c55921d3ec" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 46259 } attrs { promote: false } } peers { permanent_uuid: "65ae2553e5234a48bb69f75f1afe1ea5" member_type: NON_VOTER last_known_addr { host: "127.31.131.124" port: 42683 } attrs { promote: true } } }
I20240507 02:07:23.646210 6589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "42c4002f2beb495880b7c5db3cabab77" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "42c4002f2beb495880b7c5db3cabab77" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 40295 } } peers { permanent_uuid: "063c163f1a014ed18bce90c55921d3ec" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 46259 } attrs { promote: false } } peers { permanent_uuid: "65ae2553e5234a48bb69f75f1afe1ea5" member_type: NON_VOTER last_known_addr { host: "127.31.131.124" port: 42683 } attrs { promote: true } } }
I20240507 02:07:23.646281 6589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec [sys.catalog]: This master's current role is: FOLLOWER
I20240507 02:07:23.649852 6604 master.cc:552] Master@127.31.131.124:42683 shutdown complete.
I20240507 02:07:23.649897 6604 master_runner.cc:305] Clearing existing system tablet
I20240507 02:07:23.650022 6604 fs_manager.cc:359] Metadata directory not provided
I20240507 02:07:23.650063 6604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:07:23.650703 6604 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20240507 02:07:23.651172 6702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:23.651305 6604 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:07:23.651360 6604 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:23.651381 6604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/master-2/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/master-2/wal
uuid: "65ae2553e5234a48bb69f75f1afe1ea5"
format_stamp: "Formatted at 2024-05-07 02:07:23 on dist-test-slave-v0bg"
server_key: "c9aab099fe267ddf76e6f5ea27842e87"
server_key_iv: "348fec3514c65d75987bee9f44f071ee"
server_key_version: "encryptionkey@0"
I20240507 02:07:23.652014 6604 ts_tablet_manager.cc:1875] T 00000000000000000000000000000000 P 65ae2553e5234a48bb69f75f1afe1ea5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 02:07:23.653159 6604 ts_tablet_manager.cc:1888] T 00000000000000000000000000000000 P 65ae2553e5234a48bb69f75f1afe1ea5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20240507 02:07:23.653218 6604 log.cc:1196] T 00000000000000000000000000000000 P 65ae2553e5234a48bb69f75f1afe1ea5: Deleting WAL directory at /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20240507 02:07:23.653375 6604 ts_tablet_manager.cc:1909] T 00000000000000000000000000000000 P 65ae2553e5234a48bb69f75f1afe1ea5: Deleting consensus metadata
I20240507 02:07:23.653581 6604 master_runner.cc:315] Copying system tablet from 127.31.131.126:40295
I20240507 02:07:23.654181 6604 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 65ae2553e5234a48bb69f75f1afe1ea5: tablet copy: Beginning tablet copy session from remote peer at address 127.31.131.126:40295
I20240507 02:07:23.660513 5991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:23.654470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49796 (local address 127.31.131.126:40295)
0507 02:07:23.654510 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:23.654512 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:23.654622 (+ 110us) server_negotiation.cc:407] Connection header received
0507 02:07:23.654719 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:23.654720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:23.654730 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:23.654781 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:23.655239 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.655941 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:23.656681 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.657182 (+ 501us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:23.657868 (+ 686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:23.657871 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:23.657871 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:07:23.657876 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:23.659563 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:23.659901 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:23.659902 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:23.659903 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:23.659926 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:23.660070 (+ 144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:23.660071 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:23.660071 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:23.660168 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:23.660226 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:23.660358 (+ 132us) server_negotiation.cc:299] Negotiation successful
0507 02:07:23.660434 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20240507 02:07:23.660849 6708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:23.654417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.0.0.1:49796)
0507 02:07:23.654534 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:23.654543 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:23.654649 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:23.654783 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:23.654784 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:23.654959 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:07:23.655167 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:23.655169 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.656028 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:23.656029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:23.656605 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:23.656607 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.656684 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:23.657450 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:07:23.657455 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:07:23.657786 (+ 331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:07:23.659647 (+ 1861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:23.659648 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:23.659649 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:23.659828 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:23.659989 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:23.659990 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:23.659990 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:23.660021 (+ 31us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:23.660221 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:07:23.660222 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:07:23.660264 (+ 42us) client_negotiation.cc:769] Sending connection context
0507 02:07:23.660331 (+ 67us) client_negotiation.cc:240] Negotiation successful
0507 02:07:23.660462 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":68,"threads_started":1}
I20240507 02:07:23.661069 5971 tablet_copy_service.cc:140] P 42c4002f2beb495880b7c5db3cabab77: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 65ae2553e5234a48bb69f75f1afe1ea5 ({username='oryx', principal='oryx/127.31.131.124@KRBTEST.COM'} at 127.0.0.1:49796)
I20240507 02:07:23.661127 5971 tablet_copy_service.cc:161] P 42c4002f2beb495880b7c5db3cabab77: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 65ae2553e5234a48bb69f75f1afe1ea5 at {username='oryx', principal='oryx/127.31.131.124@KRBTEST.COM'} at 127.0.0.1:49796: session id = 65ae2553e5234a48bb69f75f1afe1ea5-00000000000000000000000000000000
I20240507 02:07:23.661674 5971 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 42c4002f2beb495880b7c5db3cabab77: Tablet Copy: opened 0 blocks and 1 log segments
I20240507 02:07:23.662077 6604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20240507 02:07:23.663200 6604 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 65ae2553e5234a48bb69f75f1afe1ea5: tablet copy: Starting download of 0 data blocks...
I20240507 02:07:23.663305 6604 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 65ae2553e5234a48bb69f75f1afe1ea5: tablet copy: Starting download of 1 WAL segments...
I20240507 02:07:23.664163 6604 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 65ae2553e5234a48bb69f75f1afe1ea5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240507 02:07:23.666756 5971 tablet_copy_service.cc:342] P 42c4002f2beb495880b7c5db3cabab77: Request end of tablet copy session 65ae2553e5234a48bb69f75f1afe1ea5-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.31.131.124@KRBTEST.COM'} at 127.0.0.1:49796
I20240507 02:07:23.666823 5971 tablet_copy_service.cc:433] P 42c4002f2beb495880b7c5db3cabab77: ending tablet copy session 65ae2553e5234a48bb69f75f1afe1ea5-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 65ae2553e5234a48bb69f75f1afe1ea5
I20240507 02:07:23.678623 6604 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:07:23.679704 6709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:07:23.679888 6604 server_base.cc:1034] running on GCE node
W20240507 02:07:23.679937 6712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:07:23.679962 6710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:07:23.680316 6604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:07:23.680469 6604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:07:23.681605 6604 hybrid_clock.cc:648] HybridClock initialized: now 1715047643681612 us; error 38 us; skew 500 ppm
May 07 02:07:23 dist-test-slave-v0bg krb5kdc[5906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047643, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.131.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:07:23.683230 6604 init.cc:377] Logged in from keytab as oryx/127.31.131.124@KRBTEST.COM (short username oryx)
I20240507 02:07:23.683493 6604 webserver.cc:460] Webserver started at http://127.31.131.124:40529/ using document root <none> and password file <none>
I20240507 02:07:23.683601 6604 fs_manager.cc:359] Metadata directory not provided
I20240507 02:07:23.683640 6604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:07:23.684319 6604 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20240507 02:07:23.684867 6719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:23.685042 6604 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240507 02:07:23.685098 6604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/master-2/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/master-2/wal
uuid: "65ae2553e5234a48bb69f75f1afe1ea5"
format_stamp: "Formatted at 2024-05-07 02:07:23 on dist-test-slave-v0bg"
server_key: "c9aab099fe267ddf76e6f5ea27842e87"
server_key_iv: "348fec3514c65d75987bee9f44f071ee"
server_key_version: "encryptionkey@0"
I20240507 02:07:23.685155 6604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:07:23.703146 6604 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:07:23.705016 6604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:07:23.707844 6604 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.124:42683
I20240507 02:07:23.708191 6604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/master-2/data/info.pb
I20240507 02:07:23.710698 6771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.124:42683 every 8 connection(s)
I20240507 02:07:23.711385 6772 sys_catalog.cc:263] Verifying existing consensus state
I20240507 02:07:23.711854 6772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65ae2553e5234a48bb69f75f1afe1ea5: Bootstrap starting.
I20240507 02:07:23.714758 6772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65ae2553e5234a48bb69f75f1afe1ea5: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240507 02:07:23.714962 6772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65ae2553e5234a48bb69f75f1afe1ea5: Bootstrap complete.
I20240507 02:07:23.715145 6772 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65ae2553e5234a48bb69f75f1afe1ea5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "42c4002f2beb495880b7c5db3cabab77" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 40295 } } peers { permanent_uuid: "063c163f1a014ed18bce90c55921d3ec" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 46259 } attrs { promote: false } } peers { permanent_uuid: "65ae2553e5234a48bb69f75f1afe1ea5" member_type: NON_VOTER last_known_addr { host: "127.31.131.124" port: 42683 } attrs { promote: true } }
I20240507 02:07:23.715229 6772 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65ae2553e5234a48bb69f75f1afe1ea5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 65ae2553e5234a48bb69f75f1afe1ea5, State: Initialized, Role: LEARNER
I20240507 02:07:23.715262 6772 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65ae2553e5234a48bb69f75f1afe1ea5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "42c4002f2beb495880b7c5db3cabab77" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 40295 } } peers { permanent_uuid: "063c163f1a014ed18bce90c55921d3ec" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 46259 } attrs { promote: false } } peers { permanent_uuid: "65ae2553e5234a48bb69f75f1afe1ea5" member_type: NON_VOTER last_known_addr { host: "127.31.131.124" port: 42683 } attrs { promote: true } }
I20240507 02:07:23.715413 6772 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65ae2553e5234a48bb69f75f1afe1ea5 [sys.catalog]: configured and running, proceeding with master startup.
I20240507 02:07:23.715432 6778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65ae2553e5234a48bb69f75f1afe1ea5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "42c4002f2beb495880b7c5db3cabab77" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 40295 } } peers { permanent_uuid: "063c163f1a014ed18bce90c55921d3ec" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 46259 } attrs { promote: false } } peers { permanent_uuid: "65ae2553e5234a48bb69f75f1afe1ea5" member_type: NON_VOTER last_known_addr { host: "127.31.131.124" port: 42683 } attrs { promote: true } } }
I20240507 02:07:23.715621 6778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65ae2553e5234a48bb69f75f1afe1ea5 [sys.catalog]: This master's current role is: LEARNER
W20240507 02:07:23.716413 6697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:07:23.716619 6697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:07:23.716653 6697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240507 02:07:23.717358 6789 catalog_manager.cc:1259] Loaded cluster ID: 4b358b1693354b3488e6edbc34acd7c3
I20240507 02:07:23.717419 6789 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 65ae2553e5234a48bb69f75f1afe1ea5: loading cluster ID for follower catalog manager: success
W20240507 02:07:23.718305 6697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:07:23.718348 6697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:07:23.718382 6697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.65
I20240507 02:07:23.718686 6789 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 65ae2553e5234a48bb69f75f1afe1ea5: acquiring CA information for follower catalog manager: success
I20240507 02:07:23.718966 6789 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 65ae2553e5234a48bb69f75f1afe1ea5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240507 02:07:23.720005 6697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:42939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/krb5kdc/oryx_127.31.131.65.keytab
--principal=oryx/127.31.131.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.65:38131
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.131.65
--webserver_port=45227
--tserver_master_addrs=127.31.131.126:40295,127.31.131.125:46259,127.31.131.124:42683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.65
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:07:23.720191 6697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:07:23.720355 6697 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:07:23.723933 6792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:07:23.723937 6794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:07:23.724148 6791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:07:23.724748 6697 server_base.cc:1034] running on GCE node
I20240507 02:07:23.724902 6697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:07:23.725078 6697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:07:23.726213 6697 hybrid_clock.cc:648] HybridClock initialized: now 1715047643726202 us; error 28 us; skew 500 ppm
May 07 02:07:23 dist-test-slave-v0bg krb5kdc[5906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047643, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.131.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:07:23.728331 6697 init.cc:377] Logged in from keytab as oryx/127.31.131.65@KRBTEST.COM (short username oryx)
I20240507 02:07:23.728669 6697 webserver.cc:460] Webserver started at http://127.31.131.65:45227/ using document root <none> and password file <none>
I20240507 02:07:23.728822 6697 fs_manager.cc:359] Metadata directory not provided
I20240507 02:07:23.728874 6697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:07:23.730010 6697 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20240507 02:07:23.730497 6801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:23.730675 6697 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:23.730742 6697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-0/wal
uuid: "081c850177844c7696632cea1ad6eb8a"
format_stamp: "Formatted at 2024-05-07 02:07:22 on dist-test-slave-v0bg"
server_key: "747fb2588ffe186983dbb85df38bb853"
server_key_iv: "1cffad56ba804b397e9afd72d1efaae7"
server_key_version: "encryptionkey@0"
I20240507 02:07:23.730980 6697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:07:23.752202 6697 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:07:23.753549 6697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:07:23.753659 6697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:07:23.753818 6697 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:07:23.754060 6697 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:07:23.754089 6697 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:23.754114 6697 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:07:23.754142 6697 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:23.758811 6697 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.65:38131
I20240507 02:07:23.758849 6914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.65:38131 every 8 connection(s)
I20240507 02:07:23.759106 6697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
May 07 02:07:23 dist-test-slave-v0bg krb5kdc[5906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047643, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.131.65@KRBTEST.COM for oryx/127.31.131.124@KRBTEST.COM
May 07 02:07:23 dist-test-slave-v0bg krb5kdc[5906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047643, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.131.65@KRBTEST.COM for oryx/127.31.131.125@KRBTEST.COM
May 07 02:07:23 dist-test-slave-v0bg krb5kdc[5906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047643, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.131.65@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM
I20240507 02:07:23.766705 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 6697
I20240507 02:07:23.766837 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 6148
I20240507 02:07:23.768043 6921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:23.759732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.124:42683 (local address 127.31.131.65:40097)
0507 02:07:23.760037 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:23.760045 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:23.760311 (+ 266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:23.760522 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:23.760525 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:23.760809 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:07:23.761084 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:23.761087 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.762232 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:23.762233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:23.762685 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:23.762686 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.762745 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:23.763155 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:07:23.763174 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:07:23.764432 (+ 1258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:07:23.766154 (+ 1722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:23.766159 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:23.766162 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:23.766434 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:23.766638 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:23.766639 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:23.766639 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:23.766667 (+ 28us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:23.767542 (+ 875us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:07:23.767547 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:07:23.767648 (+ 101us) client_negotiation.cc:769] Sending connection context
0507 02:07:23.767732 (+ 84us) client_negotiation.cc:240] Negotiation successful
0507 02:07:23.767858 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":60,"threads_started":1}
I20240507 02:07:23.768059 6601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:23.759895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:54183 (local address 127.31.131.125:46259)
0507 02:07:23.759935 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:23.759938 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:23.759947 (+ 9us) server_negotiation.cc:407] Connection header received
0507 02:07:23.760428 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:23.760429 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:23.760440 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:23.760467 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:23.761359 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.761805 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:23.762441 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.762642 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:23.764809 (+ 2167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:23.764813 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:23.764814 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:07:23.764821 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:23.766172 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:23.766475 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:23.766476 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:23.766476 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:23.766497 (+ 21us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:23.766709 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:23.766710 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:23.766710 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:23.766820 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:23.766860 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:23.767889 (+ 1029us) server_negotiation.cc:299] Negotiation successful
0507 02:07:23.767972 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20240507 02:07:23.768342 6922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:23.759894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:40097 (local address 127.31.131.124:42683)
0507 02:07:23.760116 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:23.760118 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:23.760124 (+ 6us) server_negotiation.cc:407] Connection header received
0507 02:07:23.760390 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:23.760391 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:23.760402 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:23.760469 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:23.761167 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.762168 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:23.762752 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.762921 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:23.764515 (+ 1594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:23.764520 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:23.764520 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:07:23.764526 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:23.766087 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:23.766518 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:23.766519 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:23.766519 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:23.766543 (+ 24us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:23.766728 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:23.766728 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:23.766729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:23.767476 (+ 747us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:23.767539 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:23.768173 (+ 634us) server_negotiation.cc:299] Negotiation successful
0507 02:07:23.768259 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":65,"threads_started":1}
I20240507 02:07:23.768680 6915 heartbeater.cc:347] Connected to a master server at 127.31.131.124:42683
I20240507 02:07:23.768765 6915 heartbeater.cc:464] Registering TS with master...
I20240507 02:07:23.768810 5991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:23.759683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:53855 (local address 127.31.131.126:40295)
0507 02:07:23.759726 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:23.759729 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:23.759926 (+ 197us) server_negotiation.cc:407] Connection header received
0507 02:07:23.760423 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:23.760425 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:23.760438 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:23.760467 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:23.761237 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.761940 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:23.762618 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.762785 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:23.765315 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:23.765319 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:23.765319 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:07:23.765326 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:23.766940 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:23.767748 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:23.767749 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:23.767749 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:23.767773 (+ 24us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:23.767930 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:23.767931 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:23.767931 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:23.768023 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:23.768061 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:23.768652 (+ 591us) server_negotiation.cc:299] Negotiation successful
0507 02:07:23.768728 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20240507 02:07:23.768993 6915 heartbeater.cc:511] Master 127.31.131.124:42683 requested a full tablet report, sending...
I20240507 02:07:23.769204 6920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:23.759666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.65:53855)
0507 02:07:23.759811 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:23.759836 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:23.760312 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:23.760540 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:23.760540 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:23.760826 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:07:23.761139 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:23.761142 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.762016 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:23.762017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:23.762500 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:23.762502 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.762832 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:23.763311 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:07:23.763315 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:07:23.765244 (+ 1929us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:07:23.767525 (+ 2281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:23.767526 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:23.767526 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:23.767677 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:23.767839 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:23.767840 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:23.767841 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:23.767867 (+ 26us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:23.768516 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:07:23.768516 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:07:23.768555 (+ 39us) client_negotiation.cc:769] Sending connection context
0507 02:07:23.768626 (+ 71us) client_negotiation.cc:240] Negotiation successful
0507 02:07:23.768726 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":70,"threads_started":1}
I20240507 02:07:23.769208 6919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:23.759666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.125:46259 (local address 127.31.131.65:54183)
0507 02:07:23.759860 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:23.759868 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:23.760300 (+ 432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:23.760540 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:23.760540 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:23.761112 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:07:23.761287 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:23.761288 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.761876 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:23.761876 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:23.762353 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:23.762354 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.762487 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:23.763311 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:07:23.763315 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:07:23.764693 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:07:23.766231 (+ 1538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:23.766232 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:23.766232 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:23.766415 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:23.766539 (+ 124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:23.766539 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:23.766540 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:23.766635 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:23.766901 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:07:23.767745 (+ 844us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:07:23.767779 (+ 34us) client_negotiation.cc:769] Sending connection context
0507 02:07:23.768267 (+ 488us) client_negotiation.cc:240] Negotiation successful
0507 02:07:23.768353 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":89,"threads_started":1}
I20240507 02:07:23.769513 6736 ts_manager.cc:194] Registered new tserver with Master: 081c850177844c7696632cea1ad6eb8a (127.31.131.65:38131)
I20240507 02:07:23.769711 6916 heartbeater.cc:347] Connected to a master server at 127.31.131.126:40295
I20240507 02:07:23.769762 6916 heartbeater.cc:464] Registering TS with master...
I20240507 02:07:23.769793 6917 heartbeater.cc:347] Connected to a master server at 127.31.131.125:46259
I20240507 02:07:23.769846 6917 heartbeater.cc:464] Registering TS with master...
I20240507 02:07:23.769852 6916 heartbeater.cc:511] Master 127.31.131.126:40295 requested a full tablet report, sending...
I20240507 02:07:23.769944 6917 heartbeater.cc:511] Master 127.31.131.125:46259 requested a full tablet report, sending...
I20240507 02:07:23.770121 5950 ts_manager.cc:194] Re-registered known tserver with Master: 081c850177844c7696632cea1ad6eb8a (127.31.131.65:38131)
I20240507 02:07:23.770241 6550 ts_manager.cc:194] Registered new tserver with Master: 081c850177844c7696632cea1ad6eb8a (127.31.131.65:38131)
I20240507 02:07:23.770536 5950 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.131.65@KRBTEST.COM'} at 127.31.131.65:53855
I20240507 02:07:23.772246 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.66:37593
--local_ip_for_outbound_sockets=127.31.131.66
--tserver_master_addrs=127.31.131.126:40295
--webserver_port=36661
--webserver_interface=127.31.131.66
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/krb5kdc/oryx_127.31.131.66.keytab
--principal=oryx/127.31.131.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:42939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.31.131.126:40295,127.31.131.125:46259
--tserver_master_addrs=127.31.131.126:40295,127.31.131.125:46259,127.31.131.124:42683 with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:07:23.846832 6923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:07:23.846985 6923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:07:23.847008 6923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:07:23.848403 6923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:07:23.848443 6923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:07:23.848474 6923 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.66
I20240507 02:07:23.850090 6923 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:42939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/krb5kdc/oryx_127.31.131.66.keytab
--principal=oryx/127.31.131.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.66:37593
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.131.66
--webserver_port=36661
--tserver_master_addrs=127.31.131.126:40295,127.31.131.125:46259,127.31.131.124:42683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.66
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:07:23.850265 6923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:07:23.850421 6923 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:07:23.854068 6929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:07:23.854079 6931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:07:23.854074 6928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:07:23.854586 6923 server_base.cc:1034] running on GCE node
I20240507 02:07:23.854724 6923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:07:23.854888 6923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:07:23.856016 6923 hybrid_clock.cc:648] HybridClock initialized: now 1715047643855994 us; error 33 us; skew 500 ppm
May 07 02:07:23 dist-test-slave-v0bg krb5kdc[5906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047643, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.131.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:07:23.862387 6923 init.cc:377] Logged in from keytab as oryx/127.31.131.66@KRBTEST.COM (short username oryx)
I20240507 02:07:23.862758 6923 webserver.cc:460] Webserver started at http://127.31.131.66:36661/ using document root <none> and password file <none>
I20240507 02:07:23.862900 6923 fs_manager.cc:359] Metadata directory not provided
I20240507 02:07:23.862942 6923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:07:23.863994 6923 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20240507 02:07:23.864414 6938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:23.864559 6923 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:23.864624 6923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-1/wal
uuid: "9bebe757b6734612ad65a04dd82d54f1"
format_stamp: "Formatted at 2024-05-07 02:07:22 on dist-test-slave-v0bg"
server_key: "518516610fc8c1009309e2066d4c5684"
server_key_iv: "020d854e9f9062a57befb61c36a515ef"
server_key_version: "encryptionkey@0"
I20240507 02:07:23.864902 6923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:07:23.877722 6923 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:07:23.879109 6923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:07:23.879191 6923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:07:23.879338 6923 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:07:23.879576 6923 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:07:23.879606 6923 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:23.879623 6923 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:07:23.879647 6923 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:23.884625 6923 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.66:37593
I20240507 02:07:23.884676 7051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.66:37593 every 8 connection(s)
I20240507 02:07:23.884932 6923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
I20240507 02:07:23.885852 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 6923
I20240507 02:07:23.886011 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 6282
May 07 02:07:23 dist-test-slave-v0bg krb5kdc[5906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047643, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.131.66@KRBTEST.COM for oryx/127.31.131.125@KRBTEST.COM
May 07 02:07:23 dist-test-slave-v0bg krb5kdc[5906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047643, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.131.66@KRBTEST.COM for oryx/127.31.131.124@KRBTEST.COM
May 07 02:07:23 dist-test-slave-v0bg krb5kdc[5906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047643, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.131.66@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM
I20240507 02:07:23.894083 6601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:23.885578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.66:46577 (local address 127.31.131.125:46259)
0507 02:07:23.885710 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:23.885713 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:23.885936 (+ 223us) server_negotiation.cc:407] Connection header received
0507 02:07:23.886666 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:23.886667 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:23.886679 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:23.886705 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:23.887560 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.888066 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:23.888788 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.888959 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:23.890936 (+ 1977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:23.890939 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:23.890940 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:07:23.890946 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:23.892894 (+ 1948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:23.893271 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:23.893272 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:23.893272 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:23.893294 (+ 22us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:23.893471 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:23.893472 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:23.893472 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:23.893563 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:23.893596 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:23.893917 (+ 321us) server_negotiation.cc:299] Negotiation successful
0507 02:07:23.893992 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20240507 02:07:23.894168 6922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:23.885622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.66:39697 (local address 127.31.131.124:42683)
0507 02:07:23.885670 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:23.885672 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:23.885989 (+ 317us) server_negotiation.cc:407] Connection header received
0507 02:07:23.886574 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:23.886576 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:23.886586 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:23.886641 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:23.888145 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.888595 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:23.889183 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.889345 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:23.891239 (+ 1894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:23.891243 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:23.891243 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:07:23.891250 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:23.893190 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:23.893499 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:23.893500 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:23.893500 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:23.893524 (+ 24us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:23.893685 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:23.893686 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:23.893686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:23.893779 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:23.893817 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:23.894010 (+ 193us) server_negotiation.cc:299] Negotiation successful
0507 02:07:23.894083 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20240507 02:07:23.894536 7059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:23.885520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.124:42683 (local address 127.31.131.66:39697)
0507 02:07:23.885846 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:23.885867 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:23.886486 (+ 619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:23.886722 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:23.886725 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:23.887261 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:07:23.887476 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:23.887479 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.888663 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:23.888664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:23.889115 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:23.889117 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.889176 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:23.889551 (+ 375us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:07:23.889554 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:07:23.891163 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:07:23.893270 (+ 2107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:23.893271 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:23.893272 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:23.893421 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:23.893583 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:23.893584 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:23.893584 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:23.893609 (+ 25us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:23.893859 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:07:23.893860 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:07:23.893896 (+ 36us) client_negotiation.cc:769] Sending connection context
0507 02:07:23.894269 (+ 373us) client_negotiation.cc:240] Negotiation successful
0507 02:07:23.894407 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":36,"thread_start_us":66,"threads_started":1}
I20240507 02:07:23.894541 7058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:23.885520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.125:46259 (local address 127.31.131.66:46577)
0507 02:07:23.885853 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:23.885869 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:23.886399 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:23.886768 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:23.886768 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:23.887091 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:07:23.887477 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:23.887482 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.888137 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:23.888138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:23.888713 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:23.888715 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.888837 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:23.889275 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:07:23.889294 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:07:23.890841 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:07:23.892969 (+ 2128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:23.892974 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:23.892977 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:23.893201 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:23.893348 (+ 147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:23.893348 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:23.893349 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:23.893409 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:23.893649 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:07:23.893653 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:07:23.893774 (+ 121us) client_negotiation.cc:769] Sending connection context
0507 02:07:23.894207 (+ 433us) client_negotiation.cc:240] Negotiation successful
0507 02:07:23.894341 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":3,"thread_start_us":66,"threads_started":1}
I20240507 02:07:23.894776 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.67:41427
--local_ip_for_outbound_sockets=127.31.131.67
--tserver_master_addrs=127.31.131.126:40295
--webserver_port=36947
--webserver_interface=127.31.131.67
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/krb5kdc/oryx_127.31.131.67.keytab
--principal=oryx/127.31.131.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:42939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.31.131.126:40295,127.31.131.125:46259
--tserver_master_addrs=127.31.131.126:40295,127.31.131.125:46259,127.31.131.124:42683 with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240507 02:07:23.895149 7054 heartbeater.cc:347] Connected to a master server at 127.31.131.125:46259
I20240507 02:07:23.895241 7054 heartbeater.cc:464] Registering TS with master...
I20240507 02:07:23.895417 7054 heartbeater.cc:511] Master 127.31.131.125:46259 requested a full tablet report, sending...
I20240507 02:07:23.895809 6550 ts_manager.cc:194] Registered new tserver with Master: 9bebe757b6734612ad65a04dd82d54f1 (127.31.131.66:37593)
I20240507 02:07:23.896162 7052 heartbeater.cc:347] Connected to a master server at 127.31.131.124:42683
I20240507 02:07:23.896217 7052 heartbeater.cc:464] Registering TS with master...
I20240507 02:07:23.896301 7052 heartbeater.cc:511] Master 127.31.131.124:42683 requested a full tablet report, sending...
I20240507 02:07:23.896564 6736 ts_manager.cc:194] Registered new tserver with Master: 9bebe757b6734612ad65a04dd82d54f1 (127.31.131.66:37593)
I20240507 02:07:23.897145 5991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:23.886480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.66:51787 (local address 127.31.131.126:40295)
0507 02:07:23.886530 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:23.886533 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:23.890712 (+ 4179us) server_negotiation.cc:407] Connection header received
0507 02:07:23.890804 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:23.890806 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:23.890816 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:23.890842 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:23.891359 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.891790 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:23.892402 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.892552 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:23.894449 (+ 1897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:23.894453 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:23.894454 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:07:23.894460 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:23.896199 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:23.896573 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:23.896574 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:23.896574 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:23.896598 (+ 24us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:23.896775 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:23.896776 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:23.896776 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:23.896852 (+ 76us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:23.896902 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:23.897032 (+ 130us) server_negotiation.cc:299] Negotiation successful
0507 02:07:23.897089 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20240507 02:07:23.897233 7057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:23.885519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.66:51787)
0507 02:07:23.890643 (+ 5124us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:23.890652 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:23.890737 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:23.890957 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:23.890958 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:23.891124 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:07:23.891291 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:23.891292 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.891851 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:23.891852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:23.892307 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:23.892309 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:23.892398 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:23.892749 (+ 351us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:07:23.892753 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:07:23.894049 (+ 1296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:07:23.896360 (+ 2311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:23.896360 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:23.896361 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:23.896512 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:23.896660 (+ 148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:23.896660 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:23.896661 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:23.896686 (+ 25us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:23.896907 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:07:23.896907 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:07:23.896945 (+ 38us) client_negotiation.cc:769] Sending connection context
0507 02:07:23.897017 (+ 72us) client_negotiation.cc:240] Negotiation successful
0507 02:07:23.897099 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5100,"mutex_wait_us":199,"thread_start_us":69,"threads_started":1}
I20240507 02:07:23.897552 7053 heartbeater.cc:347] Connected to a master server at 127.31.131.126:40295
I20240507 02:07:23.897605 7053 heartbeater.cc:464] Registering TS with master...
I20240507 02:07:23.897684 7053 heartbeater.cc:511] Master 127.31.131.126:40295 requested a full tablet report, sending...
I20240507 02:07:23.897909 5950 ts_manager.cc:194] Re-registered known tserver with Master: 9bebe757b6734612ad65a04dd82d54f1 (127.31.131.66:37593)
I20240507 02:07:23.898224 5950 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.131.66@KRBTEST.COM'} at 127.31.131.66:51787
W20240507 02:07:23.968634 7060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:07:23.968789 7060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:07:23.968816 7060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:07:23.970088 7060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:07:23.970126 7060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:07:23.970157 7060 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.67
I20240507 02:07:23.971536 7060 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:42939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/krb5kdc/oryx_127.31.131.67.keytab
--principal=oryx/127.31.131.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.67:41427
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.131.67
--webserver_port=36947
--tserver_master_addrs=127.31.131.126:40295,127.31.131.125:46259,127.31.131.124:42683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7060
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.67
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:07:23.971714 7060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:07:23.971854 7060 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:07:23.975327 7066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:07:23.975395 7060 server_base.cc:1034] running on GCE node
W20240507 02:07:23.975353 7065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:07:23.975504 7068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:07:23.975661 7060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:07:23.975845 7060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:07:23.976966 7060 hybrid_clock.cc:648] HybridClock initialized: now 1715047643976945 us; error 32 us; skew 500 ppm
May 07 02:07:23 dist-test-slave-v0bg krb5kdc[5906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047643, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.131.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:07:23.979046 7060 init.cc:377] Logged in from keytab as oryx/127.31.131.67@KRBTEST.COM (short username oryx)
I20240507 02:07:23.979396 7060 webserver.cc:460] Webserver started at http://127.31.131.67:36947/ using document root <none> and password file <none>
I20240507 02:07:23.979553 7060 fs_manager.cc:359] Metadata directory not provided
I20240507 02:07:23.979604 7060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:07:23.980640 7060 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20240507 02:07:23.981062 7075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:23.981199 7060 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:23.981264 7060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-2/wal
uuid: "a94723bb3109427881c73d6e66034478"
format_stamp: "Formatted at 2024-05-07 02:07:23 on dist-test-slave-v0bg"
server_key: "91d58b57379293f373856e2e8b6ac62d"
server_key_iv: "c043f9fa4e35333a63c475494d0e479a"
server_key_version: "encryptionkey@0"
I20240507 02:07:23.981489 7060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:07:23.994226 7060 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:07:23.995606 7060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:07:23.995702 7060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:07:23.995853 7060 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:07:23.996094 7060 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:07:23.996124 7060 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:23.996148 7060 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:07:23.996169 7060 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:24.000715 7060 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.67:41427
I20240507 02:07:24.000763 7189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.67:41427 every 8 connection(s)
I20240507 02:07:24.001005 7060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
May 07 02:07:24 dist-test-slave-v0bg krb5kdc[5906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047643, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.131.67@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM
May 07 02:07:24 dist-test-slave-v0bg krb5kdc[5906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047643, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.131.67@KRBTEST.COM for oryx/127.31.131.124@KRBTEST.COM
May 07 02:07:24 dist-test-slave-v0bg krb5kdc[5906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047643, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.131.67@KRBTEST.COM for oryx/127.31.131.125@KRBTEST.COM
I20240507 02:07:24.008813 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 7060
I20240507 02:07:24.009819 6601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:24.001789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.67:38475 (local address 127.31.131.125:46259)
0507 02:07:24.001829 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:24.001831 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:24.001973 (+ 142us) server_negotiation.cc:407] Connection header received
0507 02:07:24.002359 (+ 386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:24.002360 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:24.002371 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:24.002411 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:24.004021 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:24.005203 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:24.005747 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:24.005904 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:24.007302 (+ 1398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:24.007306 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:24.007306 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:07:24.007313 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:24.008659 (+ 1346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:24.009059 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:24.009060 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:24.009061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:24.009085 (+ 24us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:24.009215 (+ 130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:24.009216 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:24.009216 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:24.009309 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:24.009358 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:24.009683 (+ 325us) server_negotiation.cc:299] Negotiation successful
0507 02:07:24.009757 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20240507 02:07:24.009858 6922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:24.001658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.67:34629 (local address 127.31.131.124:42683)
0507 02:07:24.001702 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:24.001704 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:24.001981 (+ 277us) server_negotiation.cc:407] Connection header received
0507 02:07:24.002312 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:24.002313 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:24.002325 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:24.002368 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:24.003175 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:24.003932 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:24.004702 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:24.004815 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:24.007193 (+ 2378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:24.007196 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:24.007197 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:07:24.007203 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:24.008636 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:24.009007 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:24.009008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:24.009008 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:24.009029 (+ 21us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:24.009228 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:24.009228 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:24.009229 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:24.009325 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:24.009362 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:24.009681 (+ 319us) server_negotiation.cc:299] Negotiation successful
0507 02:07:24.009767 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20240507 02:07:24.009858 5991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:24.001727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.67:46643 (local address 127.31.131.126:40295)
0507 02:07:24.001778 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:24.001779 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:24.001789 (+ 10us) server_negotiation.cc:407] Connection header received
0507 02:07:24.002340 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:24.002342 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:24.002352 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:24.002379 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:24.003218 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:24.003925 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:24.004428 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:24.004599 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:24.006953 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:24.006957 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:24.006957 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:07:24.006965 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:24.008511 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:24.009004 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:24.009006 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:24.009006 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:24.009029 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:24.009228 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:24.009228 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:24.009229 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:24.009325 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:24.009362 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:24.009696 (+ 334us) server_negotiation.cc:299] Negotiation successful
0507 02:07:24.009765 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20240507 02:07:24.010275 7196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:24.001548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.125:46259 (local address 127.31.131.67:38475)
0507 02:07:24.001907 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:24.001915 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:24.002226 (+ 311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:24.002453 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:24.002453 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:24.002791 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:07:24.003150 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:24.003152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:24.005287 (+ 2135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:24.005288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:24.005674 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:24.005676 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:24.005735 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:24.006147 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:07:24.006150 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:07:24.007224 (+ 1074us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:07:24.008744 (+ 1520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:24.008745 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:24.008746 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:24.008981 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:24.009139 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:24.009140 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:24.009140 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:24.009166 (+ 26us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:24.009380 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:07:24.009386 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:07:24.009532 (+ 146us) client_negotiation.cc:769] Sending connection context
0507 02:07:24.009938 (+ 406us) client_negotiation.cc:240] Negotiation successful
0507 02:07:24.010087 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"mutex_wait_us":3,"thread_start_us":72,"threads_started":1}
I20240507 02:07:24.010279 7194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:24.001548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.124:42683 (local address 127.31.131.67:34629)
0507 02:07:24.001919 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:24.001927 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:24.002232 (+ 305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:24.002439 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:24.002442 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:24.002776 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:07:24.003064 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:24.003069 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:24.004026 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:24.004027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:24.004624 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:24.004627 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:24.004697 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:24.005093 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:07:24.005097 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:07:24.006849 (+ 1752us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:07:24.008705 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:24.008706 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:24.008706 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:24.008924 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:24.009092 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:24.009093 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:24.009093 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:24.009163 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:24.009403 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:07:24.009404 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:07:24.009528 (+ 124us) client_negotiation.cc:769] Sending connection context
0507 02:07:24.009953 (+ 425us) client_negotiation.cc:240] Negotiation successful
0507 02:07:24.010167 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":257,"thread_start_us":61,"threads_started":1}
I20240507 02:07:24.010463 7195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:24.001547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.67:46643)
0507 02:07:24.001689 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:24.001715 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:24.002222 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:24.002453 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:24.002462 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:24.002920 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:07:24.003150 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:24.003152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:24.004007 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:24.004008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:24.004356 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:24.004357 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:24.004472 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:24.004948 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:07:24.004967 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:07:24.006849 (+ 1882us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:07:24.008599 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:24.008603 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:24.008605 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:24.008924 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:24.009092 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:24.009093 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:24.009093 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:24.009163 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:24.009403 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:07:24.009404 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:07:24.009528 (+ 124us) client_negotiation.cc:769] Sending connection context
0507 02:07:24.010012 (+ 484us) client_negotiation.cc:240] Negotiation successful
0507 02:07:24.010379 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":278,"thread_start_us":58,"threads_started":1}
I20240507 02:07:24.010825 7190 heartbeater.cc:347] Connected to a master server at 127.31.131.124:42683
I20240507 02:07:24.010812 7191 heartbeater.cc:347] Connected to a master server at 127.31.131.126:40295
I20240507 02:07:24.010948 7191 heartbeater.cc:464] Registering TS with master...
I20240507 02:07:24.011161 7191 heartbeater.cc:511] Master 127.31.131.126:40295 requested a full tablet report, sending...
I20240507 02:07:24.011471 5950 ts_manager.cc:194] Re-registered known tserver with Master: a94723bb3109427881c73d6e66034478 (127.31.131.67:41427)
I20240507 02:07:24.011942 5950 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.131.67@KRBTEST.COM'} at 127.31.131.67:46643
I20240507 02:07:24.010934 7190 heartbeater.cc:464] Registering TS with master...
I20240507 02:07:24.012663 7190 heartbeater.cc:511] Master 127.31.131.124:42683 requested a full tablet report, sending...
I20240507 02:07:24.010810 7192 heartbeater.cc:347] Connected to a master server at 127.31.131.125:46259
I20240507 02:07:24.012734 7192 heartbeater.cc:464] Registering TS with master...
I20240507 02:07:24.012763 7192 heartbeater.cc:511] Master 127.31.131.125:46259 requested a full tablet report, sending...
I20240507 02:07:24.013167 6550 ts_manager.cc:194] Registered new tserver with Master: a94723bb3109427881c73d6e66034478 (127.31.131.67:41427)
I20240507 02:07:24.013350 6736 ts_manager.cc:194] Registered new tserver with Master: a94723bb3109427881c73d6e66034478 (127.31.131.67:41427)
I20240507 02:07:24.017719 6922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:24.011321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33600 (local address 127.31.131.124:42683)
0507 02:07:24.011350 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:24.011352 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:24.011685 (+ 333us) server_negotiation.cc:407] Connection header received
0507 02:07:24.011768 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:24.011769 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:24.011777 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:24.011799 (+ 22us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:24.012655 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:24.013231 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:24.013886 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:24.014045 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:24.014940 (+ 895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:24.014943 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:24.014943 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:07:24.014949 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:24.016335 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:24.016643 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:24.016645 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:24.016645 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:24.016668 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:24.016810 (+ 142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:24.016811 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:24.016811 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:24.016905 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:24.017002 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:24.017573 (+ 571us) server_negotiation.cc:299] Negotiation successful
0507 02:07:24.017638 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20240507 02:07:24.018218 5991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:24.010971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49802 (local address 127.31.131.126:40295)
0507 02:07:24.011001 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:24.011002 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:24.011638 (+ 636us) server_negotiation.cc:407] Connection header received
0507 02:07:24.011725 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:24.011726 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:24.011735 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:24.011760 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:24.012506 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:24.013151 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:24.014444 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:24.014619 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:24.015738 (+ 1119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:24.015741 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:24.015741 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:07:24.015747 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:24.017243 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:24.017534 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:24.017535 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:24.017536 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:24.017557 (+ 21us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:24.017705 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:24.017705 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:24.017706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:24.017792 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:24.017856 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:24.018042 (+ 186us) server_negotiation.cc:299] Negotiation successful
0507 02:07:24.018111 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20240507 02:07:24.018388 6601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:24.010345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42730 (local address 127.31.131.125:46259)
0507 02:07:24.010794 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:24.010795 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:24.011252 (+ 457us) server_negotiation.cc:407] Connection header received
0507 02:07:24.011332 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:24.011332 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:24.011341 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:24.011391 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:24.012281 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:24.012958 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:24.014124 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:24.014269 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:24.015790 (+ 1521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:24.015792 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:24.015793 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:07:24.015798 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:24.017376 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:24.017658 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:24.017658 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:24.017659 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:24.017680 (+ 21us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:24.017850 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:24.017852 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:24.017852 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:24.017944 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:24.018050 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:24.018253 (+ 203us) server_negotiation.cc:299] Negotiation successful
0507 02:07:24.018315 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431}
I20240507 02:07:24.019948 5950 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49802:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 02:07:24.021289 5950 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 02:07:24.027349 7210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:24.024712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.67:41427 (local address 127.0.0.1:33762)
0507 02:07:24.024823 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:24.024831 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:24.024869 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:24.025346 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:24.025347 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:24.025349 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:07:24.025520 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:24.025522 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:24.026295 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:24.026296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:24.027135 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:24.027137 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:24.027213 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:24.027217 (+ 4us) client_negotiation.cc:769] Sending connection context
0507 02:07:24.027251 (+ 34us) client_negotiation.cc:240] Negotiation successful
0507 02:07:24.027282 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":73,"threads_started":1}
I20240507 02:07:24.027439 7208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:24.024541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.65:38131 (local address 127.0.0.1:34124)
0507 02:07:24.024669 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:24.024677 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:24.024720 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:24.025195 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:24.025195 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:24.025197 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:07:24.025363 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:24.025364 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:24.026636 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:24.026637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:24.027244 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:24.027245 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:24.027307 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:24.027310 (+ 3us) client_negotiation.cc:769] Sending connection context
0507 02:07:24.027337 (+ 27us) client_negotiation.cc:240] Negotiation successful
0507 02:07:24.027374 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":82,"threads_started":1}
I20240507 02:07:24.027536 6510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:24.024449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.66:37593 (local address 127.0.0.1:43960)
0507 02:07:24.024493 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:24.024502 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:24.024541 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:24.025110 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:24.025111 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:24.025113 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:07:24.025270 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:24.025271 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:24.026579 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:24.026580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:24.027378 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:24.027380 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:24.027443 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:24.027447 (+ 4us) client_negotiation.cc:769] Sending connection context
0507 02:07:24.027470 (+ 23us) client_negotiation.cc:240] Negotiation successful
0507 02:07:24.027493 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20240507 02:07:24.027837 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:24.024877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33762 (local address 127.31.131.67:41427)
0507 02:07:24.025005 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:24.025012 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:24.025021 (+ 9us) server_negotiation.cc:407] Connection header received
0507 02:07:24.025049 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:24.025050 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:24.025065 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:24.025347 (+ 282us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:07:24.025586 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:24.026220 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:24.027215 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:24.027680 (+ 465us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:24.027721 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:24.027749 (+ 28us) server_negotiation.cc:299] Negotiation successful
0507 02:07:24.027781 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":65,"threads_started":1}
I20240507 02:07:24.028340 7211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:24.024806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34124 (local address 127.31.131.65:38131)
0507 02:07:24.024991 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:24.024999 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:24.025011 (+ 12us) server_negotiation.cc:407] Connection header received
0507 02:07:24.025050 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:24.025051 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:24.025065 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:24.025191 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:07:24.025442 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:24.026525 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:24.027678 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:24.028205 (+ 527us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:24.028234 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:24.028257 (+ 23us) server_negotiation.cc:299] Negotiation successful
0507 02:07:24.028285 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":90,"threads_started":1}
I20240507 02:07:24.028360 7123 tablet_service.cc:1467] Processing CreateTablet for tablet b1a6b9b0744c470c9daaae3e5291d321 (DEFAULT_TABLE table=test-table [id=47f495e5015647ff81141c5f550f758f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:07:24.028666 7123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1a6b9b0744c470c9daaae3e5291d321. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:07:24.028832 6849 tablet_service.cc:1467] Processing CreateTablet for tablet b1a6b9b0744c470c9daaae3e5291d321 (DEFAULT_TABLE table=test-table [id=47f495e5015647ff81141c5f550f758f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:07:24.029098 6849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1a6b9b0744c470c9daaae3e5291d321. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:07:24.030706 7209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:24.024640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43960 (local address 127.31.131.66:37593)
0507 02:07:24.024797 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:24.024805 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:24.024814 (+ 9us) server_negotiation.cc:407] Connection header received
0507 02:07:24.024984 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:24.024988 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:24.025007 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:24.025342 (+ 335us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:07:24.025408 (+ 66us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:24.026483 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:24.027857 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:24.028355 (+ 498us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:24.030591 (+ 2236us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:24.030618 (+ 27us) server_negotiation.cc:299] Negotiation successful
0507 02:07:24.030653 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":83,"threads_started":1}
I20240507 02:07:24.031826 7213 tablet_bootstrap.cc:492] T b1a6b9b0744c470c9daaae3e5291d321 P a94723bb3109427881c73d6e66034478: Bootstrap starting.
I20240507 02:07:24.032095 7214 tablet_bootstrap.cc:492] T b1a6b9b0744c470c9daaae3e5291d321 P 081c850177844c7696632cea1ad6eb8a: Bootstrap starting.
I20240507 02:07:24.032316 7213 tablet_bootstrap.cc:654] T b1a6b9b0744c470c9daaae3e5291d321 P a94723bb3109427881c73d6e66034478: Neither blocks nor log segments found. Creating new log.
I20240507 02:07:24.032444 6986 tablet_service.cc:1467] Processing CreateTablet for tablet b1a6b9b0744c470c9daaae3e5291d321 (DEFAULT_TABLE table=test-table [id=47f495e5015647ff81141c5f550f758f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:07:24.032567 7213 log.cc:824] T b1a6b9b0744c470c9daaae3e5291d321 P a94723bb3109427881c73d6e66034478: Log is configured to *not* fsync() on all Append() calls
I20240507 02:07:24.032655 7214 tablet_bootstrap.cc:654] T b1a6b9b0744c470c9daaae3e5291d321 P 081c850177844c7696632cea1ad6eb8a: Neither blocks nor log segments found. Creating new log.
I20240507 02:07:24.032701 6986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1a6b9b0744c470c9daaae3e5291d321. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:07:24.032946 7214 log.cc:824] T b1a6b9b0744c470c9daaae3e5291d321 P 081c850177844c7696632cea1ad6eb8a: Log is configured to *not* fsync() on all Append() calls
I20240507 02:07:24.033663 7214 tablet_bootstrap.cc:492] T b1a6b9b0744c470c9daaae3e5291d321 P 081c850177844c7696632cea1ad6eb8a: No bootstrap required, opened a new log
I20240507 02:07:24.033732 7214 ts_tablet_manager.cc:1367] T b1a6b9b0744c470c9daaae3e5291d321 P 081c850177844c7696632cea1ad6eb8a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240507 02:07:24.035495 7214 raft_consensus.cc:348] T b1a6b9b0744c470c9daaae3e5291d321 P 081c850177844c7696632cea1ad6eb8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bebe757b6734612ad65a04dd82d54f1" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 37593 } } peers { permanent_uuid: "081c850177844c7696632cea1ad6eb8a" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 38131 } } peers { permanent_uuid: "a94723bb3109427881c73d6e66034478" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 41427 } }
I20240507 02:07:24.035634 7214 raft_consensus.cc:374] T b1a6b9b0744c470c9daaae3e5291d321 P 081c850177844c7696632cea1ad6eb8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:07:24.035670 7214 raft_consensus.cc:729] T b1a6b9b0744c470c9daaae3e5291d321 P 081c850177844c7696632cea1ad6eb8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 081c850177844c7696632cea1ad6eb8a, State: Initialized, Role: FOLLOWER
I20240507 02:07:24.035769 7214 consensus_queue.cc:260] T b1a6b9b0744c470c9daaae3e5291d321 P 081c850177844c7696632cea1ad6eb8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bebe757b6734612ad65a04dd82d54f1" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 37593 } } peers { permanent_uuid: "081c850177844c7696632cea1ad6eb8a" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 38131 } } peers { permanent_uuid: "a94723bb3109427881c73d6e66034478" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 41427 } }
I20240507 02:07:24.036001 7214 ts_tablet_manager.cc:1398] T b1a6b9b0744c470c9daaae3e5291d321 P 081c850177844c7696632cea1ad6eb8a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240507 02:07:24.036088 6916 heartbeater.cc:503] Master 127.31.131.126:40295 was elected leader, sending a full tablet report...
I20240507 02:07:24.036283 7217 tablet_bootstrap.cc:492] T b1a6b9b0744c470c9daaae3e5291d321 P 9bebe757b6734612ad65a04dd82d54f1: Bootstrap starting.
I20240507 02:07:24.037079 7217 tablet_bootstrap.cc:654] T b1a6b9b0744c470c9daaae3e5291d321 P 9bebe757b6734612ad65a04dd82d54f1: Neither blocks nor log segments found. Creating new log.
I20240507 02:07:24.037369 7217 log.cc:824] T b1a6b9b0744c470c9daaae3e5291d321 P 9bebe757b6734612ad65a04dd82d54f1: Log is configured to *not* fsync() on all Append() calls
I20240507 02:07:24.038003 7217 tablet_bootstrap.cc:492] T b1a6b9b0744c470c9daaae3e5291d321 P 9bebe757b6734612ad65a04dd82d54f1: No bootstrap required, opened a new log
I20240507 02:07:24.038056 7217 ts_tablet_manager.cc:1367] T b1a6b9b0744c470c9daaae3e5291d321 P 9bebe757b6734612ad65a04dd82d54f1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240507 02:07:24.038827 7213 tablet_bootstrap.cc:492] T b1a6b9b0744c470c9daaae3e5291d321 P a94723bb3109427881c73d6e66034478: No bootstrap required, opened a new log
I20240507 02:07:24.038880 7213 ts_tablet_manager.cc:1367] T b1a6b9b0744c470c9daaae3e5291d321 P a94723bb3109427881c73d6e66034478: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240507 02:07:24.039260 7217 raft_consensus.cc:348] T b1a6b9b0744c470c9daaae3e5291d321 P 9bebe757b6734612ad65a04dd82d54f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bebe757b6734612ad65a04dd82d54f1" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 37593 } } peers { permanent_uuid: "081c850177844c7696632cea1ad6eb8a" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 38131 } } peers { permanent_uuid: "a94723bb3109427881c73d6e66034478" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 41427 } }
I20240507 02:07:24.039364 7217 raft_consensus.cc:374] T b1a6b9b0744c470c9daaae3e5291d321 P 9bebe757b6734612ad65a04dd82d54f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:07:24.039382 7217 raft_consensus.cc:729] T b1a6b9b0744c470c9daaae3e5291d321 P 9bebe757b6734612ad65a04dd82d54f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bebe757b6734612ad65a04dd82d54f1, State: Initialized, Role: FOLLOWER
I20240507 02:07:24.039470 7217 consensus_queue.cc:260] T b1a6b9b0744c470c9daaae3e5291d321 P 9bebe757b6734612ad65a04dd82d54f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bebe757b6734612ad65a04dd82d54f1" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 37593 } } peers { permanent_uuid: "081c850177844c7696632cea1ad6eb8a" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 38131 } } peers { permanent_uuid: "a94723bb3109427881c73d6e66034478" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 41427 } }
I20240507 02:07:24.039669 7217 ts_tablet_manager.cc:1398] T b1a6b9b0744c470c9daaae3e5291d321 P 9bebe757b6734612ad65a04dd82d54f1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240507 02:07:24.039863 7053 heartbeater.cc:503] Master 127.31.131.126:40295 was elected leader, sending a full tablet report...
I20240507 02:07:24.040269 7213 raft_consensus.cc:348] T b1a6b9b0744c470c9daaae3e5291d321 P a94723bb3109427881c73d6e66034478 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bebe757b6734612ad65a04dd82d54f1" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 37593 } } peers { permanent_uuid: "081c850177844c7696632cea1ad6eb8a" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 38131 } } peers { permanent_uuid: "a94723bb3109427881c73d6e66034478" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 41427 } }
I20240507 02:07:24.040378 7213 raft_consensus.cc:374] T b1a6b9b0744c470c9daaae3e5291d321 P a94723bb3109427881c73d6e66034478 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:07:24.040400 7213 raft_consensus.cc:729] T b1a6b9b0744c470c9daaae3e5291d321 P a94723bb3109427881c73d6e66034478 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a94723bb3109427881c73d6e66034478, State: Initialized, Role: FOLLOWER
I20240507 02:07:24.040493 7213 consensus_queue.cc:260] T b1a6b9b0744c470c9daaae3e5291d321 P a94723bb3109427881c73d6e66034478 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bebe757b6734612ad65a04dd82d54f1" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 37593 } } peers { permanent_uuid: "081c850177844c7696632cea1ad6eb8a" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 38131 } } peers { permanent_uuid: "a94723bb3109427881c73d6e66034478" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 41427 } }
I20240507 02:07:24.040721 7191 heartbeater.cc:503] Master 127.31.131.126:40295 was elected leader, sending a full tablet report...
I20240507 02:07:24.040885 7213 ts_tablet_manager.cc:1398] T b1a6b9b0744c470c9daaae3e5291d321 P a94723bb3109427881c73d6e66034478: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240507 02:07:24.050642 7218 raft_consensus.cc:482] T b1a6b9b0744c470c9daaae3e5291d321 P 081c850177844c7696632cea1ad6eb8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 02:07:24.050712 7218 raft_consensus.cc:504] T b1a6b9b0744c470c9daaae3e5291d321 P 081c850177844c7696632cea1ad6eb8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bebe757b6734612ad65a04dd82d54f1" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 37593 } } peers { permanent_uuid: "081c850177844c7696632cea1ad6eb8a" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 38131 } } peers { permanent_uuid: "a94723bb3109427881c73d6e66034478" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 41427 } }
I20240507 02:07:24.050944 7218 leader_election.cc:288] T b1a6b9b0744c470c9daaae3e5291d321 P 081c850177844c7696632cea1ad6eb8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bebe757b6734612ad65a04dd82d54f1 (127.31.131.66:37593), a94723bb3109427881c73d6e66034478 (127.31.131.67:41427)
I20240507 02:07:24.052978 6920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:24.051078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.66:37593 (local address 127.31.131.65:44567)
0507 02:07:24.051115 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:24.051125 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:24.051159 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:24.051286 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:24.051287 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:24.051289 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:07:24.051457 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:24.051458 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:24.052102 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:24.052102 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:24.052739 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:24.052742 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:24.052814 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:24.052818 (+ 4us) client_negotiation.cc:769] Sending connection context
0507 02:07:24.052851 (+ 33us) client_negotiation.cc:240] Negotiation successful
0507 02:07:24.052883 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20240507 02:07:24.053043 6919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:24.051172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.67:41427 (local address 127.31.131.65:58995)
0507 02:07:24.051207 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:24.051214 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:24.051240 (+ 26us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:24.051341 (+ 101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:24.051342 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:24.051343 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:07:24.051498 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:24.051499 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:24.052147 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:24.052148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:24.052782 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:24.052784 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:24.052842 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:24.052845 (+ 3us) client_negotiation.cc:769] Sending connection context
0507 02:07:24.052873 (+ 28us) client_negotiation.cc:240] Negotiation successful
0507 02:07:24.052905 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20240507 02:07:24.053359 7209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:24.051117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:44567 (local address 127.31.131.66:37593)
0507 02:07:24.051164 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:24.051165 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:24.051178 (+ 13us) server_negotiation.cc:407] Connection header received
0507 02:07:24.051209 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:24.051210 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:24.051219 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:24.051282 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:07:24.051521 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:24.052013 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:24.052812 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:24.053248 (+ 436us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:24.053255 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:24.053273 (+ 18us) server_negotiation.cc:299] Negotiation successful
0507 02:07:24.053302 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20240507 02:07:24.053509 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:24.051223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:58995 (local address 127.31.131.67:41427)
0507 02:07:24.051263 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:24.051265 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:24.051271 (+ 6us) server_negotiation.cc:407] Connection header received
0507 02:07:24.051289 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:24.051290 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:24.051297 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:24.051339 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:07:24.051552 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:24.052079 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:24.053005 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:24.053415 (+ 410us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:24.053421 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:24.053439 (+ 18us) server_negotiation.cc:299] Negotiation successful
0507 02:07:24.053473 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20240507 02:07:24.053647 7006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1a6b9b0744c470c9daaae3e5291d321" candidate_uuid: "081c850177844c7696632cea1ad6eb8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bebe757b6734612ad65a04dd82d54f1" is_pre_election: true
I20240507 02:07:24.053755 7006 raft_consensus.cc:2454] T b1a6b9b0744c470c9daaae3e5291d321 P 9bebe757b6734612ad65a04dd82d54f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 081c850177844c7696632cea1ad6eb8a in term 0.
I20240507 02:07:24.053758 7143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1a6b9b0744c470c9daaae3e5291d321" candidate_uuid: "081c850177844c7696632cea1ad6eb8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a94723bb3109427881c73d6e66034478" is_pre_election: true
I20240507 02:07:24.053855 7143 raft_consensus.cc:2454] T b1a6b9b0744c470c9daaae3e5291d321 P a94723bb3109427881c73d6e66034478 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 081c850177844c7696632cea1ad6eb8a in term 0.
I20240507 02:07:24.053995 6802 leader_election.cc:302] T b1a6b9b0744c470c9daaae3e5291d321 P 081c850177844c7696632cea1ad6eb8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 081c850177844c7696632cea1ad6eb8a, 9bebe757b6734612ad65a04dd82d54f1; no voters:
I20240507 02:07:24.054090 7218 raft_consensus.cc:2781] T b1a6b9b0744c470c9daaae3e5291d321 P 081c850177844c7696632cea1ad6eb8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 02:07:24.054157 7218 raft_consensus.cc:482] T b1a6b9b0744c470c9daaae3e5291d321 P 081c850177844c7696632cea1ad6eb8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 02:07:24.054173 7218 raft_consensus.cc:3037] T b1a6b9b0744c470c9daaae3e5291d321 P 081c850177844c7696632cea1ad6eb8a [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:07:24.054987 7218 raft_consensus.cc:504] T b1a6b9b0744c470c9daaae3e5291d321 P 081c850177844c7696632cea1ad6eb8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bebe757b6734612ad65a04dd82d54f1" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 37593 } } peers { permanent_uuid: "081c850177844c7696632cea1ad6eb8a" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 38131 } } peers { permanent_uuid: "a94723bb3109427881c73d6e66034478" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 41427 } }
I20240507 02:07:24.055083 7218 leader_election.cc:288] T b1a6b9b0744c470c9daaae3e5291d321 P 081c850177844c7696632cea1ad6eb8a [CANDIDATE]: Term 1 election: Requested vote from peers 9bebe757b6734612ad65a04dd82d54f1 (127.31.131.66:37593), a94723bb3109427881c73d6e66034478 (127.31.131.67:41427)
I20240507 02:07:24.055231 7006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1a6b9b0744c470c9daaae3e5291d321" candidate_uuid: "081c850177844c7696632cea1ad6eb8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bebe757b6734612ad65a04dd82d54f1"
I20240507 02:07:24.055262 7143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1a6b9b0744c470c9daaae3e5291d321" candidate_uuid: "081c850177844c7696632cea1ad6eb8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a94723bb3109427881c73d6e66034478"
I20240507 02:07:24.055292 7006 raft_consensus.cc:3037] T b1a6b9b0744c470c9daaae3e5291d321 P 9bebe757b6734612ad65a04dd82d54f1 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:07:24.055317 7143 raft_consensus.cc:3037] T b1a6b9b0744c470c9daaae3e5291d321 P a94723bb3109427881c73d6e66034478 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:07:24.055850 7006 raft_consensus.cc:2454] T b1a6b9b0744c470c9daaae3e5291d321 P 9bebe757b6734612ad65a04dd82d54f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 081c850177844c7696632cea1ad6eb8a in term 1.
I20240507 02:07:24.055861 7143 raft_consensus.cc:2454] T b1a6b9b0744c470c9daaae3e5291d321 P a94723bb3109427881c73d6e66034478 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 081c850177844c7696632cea1ad6eb8a in term 1.
I20240507 02:07:24.055996 6802 leader_election.cc:302] T b1a6b9b0744c470c9daaae3e5291d321 P 081c850177844c7696632cea1ad6eb8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 081c850177844c7696632cea1ad6eb8a, 9bebe757b6734612ad65a04dd82d54f1; no voters:
I20240507 02:07:24.056078 7218 raft_consensus.cc:2781] T b1a6b9b0744c470c9daaae3e5291d321 P 081c850177844c7696632cea1ad6eb8a [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:07:24.056217 7218 raft_consensus.cc:686] T b1a6b9b0744c470c9daaae3e5291d321 P 081c850177844c7696632cea1ad6eb8a [term 1 LEADER]: Becoming Leader. State: Replica: 081c850177844c7696632cea1ad6eb8a, State: Running, Role: LEADER
I20240507 02:07:24.056283 7218 consensus_queue.cc:237] T b1a6b9b0744c470c9daaae3e5291d321 P 081c850177844c7696632cea1ad6eb8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bebe757b6734612ad65a04dd82d54f1" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 37593 } } peers { permanent_uuid: "081c850177844c7696632cea1ad6eb8a" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 38131 } } peers { permanent_uuid: "a94723bb3109427881c73d6e66034478" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 41427 } }
I20240507 02:07:24.056926 5950 catalog_manager.cc:5484] T b1a6b9b0744c470c9daaae3e5291d321 P 081c850177844c7696632cea1ad6eb8a reported cstate change: term changed from 0 to 1, leader changed from <none> to 081c850177844c7696632cea1ad6eb8a (127.31.131.65). New cstate: current_term: 1 leader_uuid: "081c850177844c7696632cea1ad6eb8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bebe757b6734612ad65a04dd82d54f1" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 37593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "081c850177844c7696632cea1ad6eb8a" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 38131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a94723bb3109427881c73d6e66034478" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 41427 } health_report { overall_health: UNKNOWN } } }
I20240507 02:07:24.063815 7211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:24.062237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34140 (local address 127.31.131.65:38131)
0507 02:07:24.062279 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:24.062281 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:24.062299 (+ 18us) server_negotiation.cc:407] Connection header received
0507 02:07:24.062322 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:24.062323 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:24.062335 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:24.062382 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0507 02:07:24.062577 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:24.062996 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:24.063471 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:24.063570 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:24.063595 (+ 25us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0507 02:07:24.063661 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0507 02:07:24.063710 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:24.063748 (+ 38us) server_negotiation.cc:299] Negotiation successful
0507 02:07:24.063764 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20240507 02:07:24.065058 7143 raft_consensus.cc:1261] T b1a6b9b0744c470c9daaae3e5291d321 P a94723bb3109427881c73d6e66034478 [term 1 FOLLOWER]: Refusing update from remote peer 081c850177844c7696632cea1ad6eb8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 02:07:24.065081 7006 raft_consensus.cc:1261] T b1a6b9b0744c470c9daaae3e5291d321 P 9bebe757b6734612ad65a04dd82d54f1 [term 1 FOLLOWER]: Refusing update from remote peer 081c850177844c7696632cea1ad6eb8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 02:07:24.065312 7218 consensus_queue.cc:1035] T b1a6b9b0744c470c9daaae3e5291d321 P 081c850177844c7696632cea1ad6eb8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a94723bb3109427881c73d6e66034478" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 41427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 02:07:24.065428 7222 consensus_queue.cc:1035] T b1a6b9b0744c470c9daaae3e5291d321 P 081c850177844c7696632cea1ad6eb8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bebe757b6734612ad65a04dd82d54f1" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 37593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 02:07:24.069160 7227 mvcc.cc:205] Tried to move back new op lower bound from 7024835150088650752 to 7024835150055006208. Current Snapshot: MvccSnapshot[applied={T|T < 7024835150088650752 or (T in {7024835150088650752})}]
I20240507 02:07:24.070008 5950 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49802:
table { table_name: "test-table" } modify_external_catalogs: true
I20240507 02:07:24.070091 5950 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49802:
table { table_name: "test-table" } modify_external_catalogs: true
I20240507 02:07:24.071202 5950 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 42c4002f2beb495880b7c5db3cabab77: Sending DeleteTablet for 3 replicas of tablet b1a6b9b0744c470c9daaae3e5291d321
I20240507 02:07:24.071553 7123 tablet_service.cc:1514] Processing DeleteTablet for tablet b1a6b9b0744c470c9daaae3e5291d321 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:07:24 UTC) from {username='oryx', principal='oryx/127.31.131.126@KRBTEST.COM'} at 127.0.0.1:33762
I20240507 02:07:24.071753 7235 tablet_replica.cc:330] T b1a6b9b0744c470c9daaae3e5291d321 P a94723bb3109427881c73d6e66034478: stopping tablet replica
I20240507 02:07:24.071864 6986 tablet_service.cc:1514] Processing DeleteTablet for tablet b1a6b9b0744c470c9daaae3e5291d321 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:07:24 UTC) from {username='oryx', principal='oryx/127.31.131.126@KRBTEST.COM'} at 127.0.0.1:43960
I20240507 02:07:24.072081 6849 tablet_service.cc:1514] Processing DeleteTablet for tablet b1a6b9b0744c470c9daaae3e5291d321 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:07:24 UTC) from {username='oryx', principal='oryx/127.31.131.126@KRBTEST.COM'} at 127.0.0.1:34124
I20240507 02:07:24.072233 7236 tablet_replica.cc:330] T b1a6b9b0744c470c9daaae3e5291d321 P 9bebe757b6734612ad65a04dd82d54f1: stopping tablet replica
I20240507 02:07:24.072324 7236 raft_consensus.cc:2229] T b1a6b9b0744c470c9daaae3e5291d321 P 9bebe757b6734612ad65a04dd82d54f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 02:07:24.072331 7235 raft_consensus.cc:2229] T b1a6b9b0744c470c9daaae3e5291d321 P a94723bb3109427881c73d6e66034478 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 02:07:24.072453 7236 raft_consensus.cc:2258] T b1a6b9b0744c470c9daaae3e5291d321 P 9bebe757b6734612ad65a04dd82d54f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 02:07:24.072453 7235 raft_consensus.cc:2258] T b1a6b9b0744c470c9daaae3e5291d321 P a94723bb3109427881c73d6e66034478 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 02:07:24.072557 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 5919
I20240507 02:07:24.072690 7238 tablet_replica.cc:330] T b1a6b9b0744c470c9daaae3e5291d321 P 081c850177844c7696632cea1ad6eb8a: stopping tablet replica
I20240507 02:07:24.072755 7238 raft_consensus.cc:2229] T b1a6b9b0744c470c9daaae3e5291d321 P 081c850177844c7696632cea1ad6eb8a [term 1 LEADER]: Raft consensus shutting down.
I20240507 02:07:24.072816 7238 raft_consensus.cc:2258] T b1a6b9b0744c470c9daaae3e5291d321 P 081c850177844c7696632cea1ad6eb8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 02:07:24.073225 7236 ts_tablet_manager.cc:1875] T b1a6b9b0744c470c9daaae3e5291d321 P 9bebe757b6734612ad65a04dd82d54f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 02:07:24.073937 7235 ts_tablet_manager.cc:1875] T b1a6b9b0744c470c9daaae3e5291d321 P a94723bb3109427881c73d6e66034478: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 02:07:24.074712 7236 ts_tablet_manager.cc:1888] T b1a6b9b0744c470c9daaae3e5291d321 P 9bebe757b6734612ad65a04dd82d54f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240507 02:07:24.074904 7236 log.cc:1196] T b1a6b9b0744c470c9daaae3e5291d321 P 9bebe757b6734612ad65a04dd82d54f1: Deleting WAL directory at /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-1/wal/wals/b1a6b9b0744c470c9daaae3e5291d321
I20240507 02:07:24.075368 7236 ts_tablet_manager.cc:1909] T b1a6b9b0744c470c9daaae3e5291d321 P 9bebe757b6734612ad65a04dd82d54f1: Deleting consensus metadata
I20240507 02:07:24.075520 7235 ts_tablet_manager.cc:1888] T b1a6b9b0744c470c9daaae3e5291d321 P a94723bb3109427881c73d6e66034478: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240507 02:07:24.075584 7235 log.cc:1196] T b1a6b9b0744c470c9daaae3e5291d321 P a94723bb3109427881c73d6e66034478: Deleting WAL directory at /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-2/wal/wals/b1a6b9b0744c470c9daaae3e5291d321
I20240507 02:07:24.075780 7235 ts_tablet_manager.cc:1909] T b1a6b9b0744c470c9daaae3e5291d321 P a94723bb3109427881c73d6e66034478: Deleting consensus metadata
W20240507 02:07:24.078689 6942 connection.cc:535] server connection from 127.0.0.1:43960 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240507 02:07:24.078843 7078 connection.cc:441] server connection from 127.0.0.1:33762 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33762 (request call id 1) could send its response
I20240507 02:07:24.079092 7238 ts_tablet_manager.cc:1875] T b1a6b9b0744c470c9daaae3e5291d321 P 081c850177844c7696632cea1ad6eb8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 02:07:24.080382 7238 ts_tablet_manager.cc:1888] T b1a6b9b0744c470c9daaae3e5291d321 P 081c850177844c7696632cea1ad6eb8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240507 02:07:24.080447 7238 log.cc:1196] T b1a6b9b0744c470c9daaae3e5291d321 P 081c850177844c7696632cea1ad6eb8a: Deleting WAL directory at /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0/minicluster-data/ts-0/wal/wals/b1a6b9b0744c470c9daaae3e5291d321
I20240507 02:07:24.080694 7238 ts_tablet_manager.cc:1909] T b1a6b9b0744c470c9daaae3e5291d321 P 081c850177844c7696632cea1ad6eb8a: Deleting consensus metadata
W20240507 02:07:24.080988 6803 connection.cc:441] server connection from 127.0.0.1:34124 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34124 (request call id 1) could send its response
I20240507 02:07:24.088362 6601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:24.079964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42734 (local address 127.31.131.125:46259)
0507 02:07:24.080014 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:24.080016 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:24.080207 (+ 191us) server_negotiation.cc:407] Connection header received
0507 02:07:24.080306 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:24.080307 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:24.080318 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:24.080369 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:24.081047 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:24.081521 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:24.082025 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:24.082181 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:24.084799 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:24.084804 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:24.084804 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:07:24.084811 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:24.087270 (+ 2459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:24.087645 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:24.087647 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:24.087647 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:24.087671 (+ 24us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:24.087824 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:24.087825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:24.087825 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:24.087927 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:24.088030 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:24.088199 (+ 169us) server_negotiation.cc:299] Negotiation successful
0507 02:07:24.088271 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20240507 02:07:24.089586 7246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:24.080121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33628 (local address 127.31.131.124:42683)
0507 02:07:24.080205 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:24.080207 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:24.080233 (+ 26us) server_negotiation.cc:407] Connection header received
0507 02:07:24.080424 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:24.080425 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:24.080435 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:24.080489 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:24.081267 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:24.082036 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:24.083119 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:24.084623 (+ 1504us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:24.085648 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:24.085652 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:24.085652 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:07:24.085659 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:24.088179 (+ 2520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:24.088837 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:24.088838 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:24.088839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:24.088862 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:24.089018 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:24.089019 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:24.089019 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:24.089157 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:24.089238 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:24.089418 (+ 180us) server_negotiation.cc:299] Negotiation successful
0507 02:07:24.089495 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70,"thread_start_us":43,"threads_started":1}
I20240507 02:07:24.090200 6922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:24.072096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33614 (local address 127.31.131.124:42683)
0507 02:07:24.072136 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:24.072138 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:24.082725 (+ 10587us) server_negotiation.cc:407] Connection header received
0507 02:07:24.082969 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:24.082972 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:24.083017 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:24.083144 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:24.085064 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:24.085766 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:24.086410 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:24.086592 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:24.087570 (+ 978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:24.087573 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:24.087574 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:07:24.087580 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:24.089184 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:24.089501 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:24.089502 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:24.089502 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:24.089522 (+ 20us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:24.089693 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:24.089694 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:24.089694 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:24.089783 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:24.089849 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:24.090050 (+ 201us) server_negotiation.cc:299] Negotiation successful
0507 02:07:24.090118 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20240507 02:07:25.068882 7247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:25.068627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.67:51699)
0507 02:07:25.068754 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:25.068823 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":71,"threads_started":1}
I20240507 02:07:25.069378 7248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:25.068940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.65:41211)
0507 02:07:25.069276 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:25.069332 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":61,"threads_started":1}
W20240507 02:07:25.069391 7191 heartbeater.cc:644] Failed to heartbeat to 127.31.131.126:40295 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
I20240507 02:07:25.069463 7249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:25.069178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.66:59207)
0507 02:07:25.069383 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:25.069423 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":46,"threads_started":1}
W20240507 02:07:25.069535 6916 heartbeater.cc:644] Failed to heartbeat to 127.31.131.126:40295 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
W20240507 02:07:25.069562 7053 heartbeater.cc:644] Failed to heartbeat to 127.31.131.126:40295 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
I20240507 02:07:25.572130 7250 raft_consensus.cc:482] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 42c4002f2beb495880b7c5db3cabab77)
I20240507 02:07:25.572284 7250 raft_consensus.cc:504] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "42c4002f2beb495880b7c5db3cabab77" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 40295 } } peers { permanent_uuid: "063c163f1a014ed18bce90c55921d3ec" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 46259 } attrs { promote: false } } peers { permanent_uuid: "65ae2553e5234a48bb69f75f1afe1ea5" member_type: NON_VOTER last_known_addr { host: "127.31.131.124" port: 42683 } attrs { promote: true } }
I20240507 02:07:25.572536 7250 leader_election.cc:288] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42c4002f2beb495880b7c5db3cabab77 (127.31.131.126:40295)
I20240507 02:07:25.572860 7251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:25.572673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.0.0.1:50284)
0507 02:07:25.572770 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:25.572820 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79,"thread_start_us":42,"threads_started":1}
W20240507 02:07:25.572957 6536 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
I20240507 02:07:25.573293 7251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:25.573207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.0.0.1:50300)
0507 02:07:25.573252 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:25.573276 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20240507 02:07:25.573367 6536 leader_election.cc:334] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 42c4002f2beb495880b7c5db3cabab77 (127.31.131.126:40295): Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
I20240507 02:07:25.573406 6536 leader_election.cc:302] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 063c163f1a014ed18bce90c55921d3ec; no voters: 42c4002f2beb495880b7c5db3cabab77
I20240507 02:07:25.573494 7250 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 02:07:26.070114 7253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:26.069689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.67:45439)
0507 02:07:26.070016 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:26.070054 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":81,"threads_started":1}
I20240507 02:07:26.070335 7255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:26.069888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.65:48577)
0507 02:07:26.070252 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:26.070287 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":59,"threads_started":1}
I20240507 02:07:26.070362 7254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:26.069836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.66:43917)
0507 02:07:26.070290 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:26.070324 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":53,"threads_started":1}
I20240507 02:07:27.071015 7256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:27.070632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.67:44755)
0507 02:07:27.070815 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:27.070856 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":70,"threads_started":1}
I20240507 02:07:27.071054 7257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:27.070867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.65:46177)
0507 02:07:27.070963 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:27.071001 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77,"thread_start_us":55,"threads_started":1}
I20240507 02:07:27.071544 7258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:27.071354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.66:56859)
0507 02:07:27.071471 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:27.071500 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":64,"threads_started":1}
I20240507 02:07:27.417515 7259 raft_consensus.cc:482] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 42c4002f2beb495880b7c5db3cabab77)
I20240507 02:07:27.417601 7259 raft_consensus.cc:504] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "42c4002f2beb495880b7c5db3cabab77" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 40295 } } peers { permanent_uuid: "063c163f1a014ed18bce90c55921d3ec" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 46259 } attrs { promote: false } } peers { permanent_uuid: "65ae2553e5234a48bb69f75f1afe1ea5" member_type: NON_VOTER last_known_addr { host: "127.31.131.124" port: 42683 } attrs { promote: true } }
I20240507 02:07:27.417753 7259 leader_election.cc:288] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42c4002f2beb495880b7c5db3cabab77 (127.31.131.126:40295)
I20240507 02:07:27.418112 7260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:27.417925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.0.0.1:50582)
0507 02:07:27.418039 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:27.418075 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":60,"threads_started":1}
I20240507 02:07:27.418509 7260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:27.418420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.0.0.1:50592)
0507 02:07:27.418463 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:27.418491 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20240507 02:07:27.418591 6536 leader_election.cc:334] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 42c4002f2beb495880b7c5db3cabab77 (127.31.131.126:40295): Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
I20240507 02:07:27.418630 6536 leader_election.cc:302] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 063c163f1a014ed18bce90c55921d3ec; no voters: 42c4002f2beb495880b7c5db3cabab77
I20240507 02:07:27.418696 7259 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 02:07:28.071902 7262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:28.071599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.67:53155)
0507 02:07:28.071809 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:28.071843 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":73,"threads_started":1}
I20240507 02:07:28.072034 7263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:28.071758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.65:35669)
0507 02:07:28.071965 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:28.071993 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":50,"threads_started":1}
I20240507 02:07:28.072206 7264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:28.071891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.66:53487)
0507 02:07:28.072138 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:28.072167 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":44,"threads_started":1}
I20240507 02:07:29.072558 7265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:29.072344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.65:51901)
0507 02:07:29.072469 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:29.072499 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":62,"threads_started":1}
I20240507 02:07:29.072659 7266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:29.072349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.67:52159)
0507 02:07:29.072579 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:29.072611 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":175,"threads_started":1}
I20240507 02:07:29.072769 7267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:29.072610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.66:43453)
0507 02:07:29.072700 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:29.072727 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77,"thread_start_us":43,"threads_started":1}
I20240507 02:07:29.957582 7268 raft_consensus.cc:482] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 42c4002f2beb495880b7c5db3cabab77)
I20240507 02:07:29.957688 7268 raft_consensus.cc:504] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "42c4002f2beb495880b7c5db3cabab77" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 40295 } } peers { permanent_uuid: "063c163f1a014ed18bce90c55921d3ec" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 46259 } attrs { promote: false } } peers { permanent_uuid: "65ae2553e5234a48bb69f75f1afe1ea5" member_type: NON_VOTER last_known_addr { host: "127.31.131.124" port: 42683 } attrs { promote: true } }
I20240507 02:07:29.957840 7268 leader_election.cc:288] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42c4002f2beb495880b7c5db3cabab77 (127.31.131.126:40295)
I20240507 02:07:29.958127 7269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:29.957971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.0.0.1:50838)
0507 02:07:29.958057 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:29.958087 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70,"thread_start_us":42,"threads_started":1}
I20240507 02:07:29.958494 7269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:29.958407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.0.0.1:50842)
0507 02:07:29.958449 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:29.958478 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20240507 02:07:29.958590 6536 leader_election.cc:334] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 42c4002f2beb495880b7c5db3cabab77 (127.31.131.126:40295): Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
I20240507 02:07:29.958631 6536 leader_election.cc:302] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 063c163f1a014ed18bce90c55921d3ec; no voters: 42c4002f2beb495880b7c5db3cabab77
I20240507 02:07:29.958701 7268 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 02:07:30.073124 7271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:30.072891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.65:58043)
0507 02:07:30.073027 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:30.073065 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":71,"threads_started":1}
I20240507 02:07:30.073272 7272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:30.073063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.66:60761)
0507 02:07:30.073189 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:30.073222 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":72,"threads_started":1}
I20240507 02:07:30.073311 7273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:30.073093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.67:33315)
0507 02:07:30.073188 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:30.073272 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":48,"threads_started":1}
I20240507 02:07:31.073733 7274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:31.073475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.65:41123)
0507 02:07:31.073608 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:31.073657 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":70,"threads_started":1}
I20240507 02:07:31.073786 7275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:31.073594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.66:34391)
0507 02:07:31.073703 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:31.073734 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76,"thread_start_us":49,"threads_started":1}
I20240507 02:07:31.073932 7276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:31.073634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.67:48387)
0507 02:07:31.073804 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:31.073835 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":67,"threads_started":1}
I20240507 02:07:32.074383 7278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:32.074176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.65:40917)
0507 02:07:32.074287 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:32.074317 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":51,"threads_started":1}
I20240507 02:07:32.074515 7279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:32.074322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.67:58003)
0507 02:07:32.074425 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:32.074454 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":51,"threads_started":1}
I20240507 02:07:32.074517 7277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:32.074100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.66:43415)
0507 02:07:32.074408 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:32.074438 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":69,"threads_started":1}
I20240507 02:07:32.996381 7280 raft_consensus.cc:482] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 42c4002f2beb495880b7c5db3cabab77)
I20240507 02:07:32.996479 7280 raft_consensus.cc:504] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "42c4002f2beb495880b7c5db3cabab77" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 40295 } } peers { permanent_uuid: "063c163f1a014ed18bce90c55921d3ec" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 46259 } attrs { promote: false } } peers { permanent_uuid: "65ae2553e5234a48bb69f75f1afe1ea5" member_type: NON_VOTER last_known_addr { host: "127.31.131.124" port: 42683 } attrs { promote: true } }
I20240507 02:07:32.996621 7280 leader_election.cc:288] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42c4002f2beb495880b7c5db3cabab77 (127.31.131.126:40295)
I20240507 02:07:32.996918 7282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:32.996755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.0.0.1:39406)
0507 02:07:32.996846 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:32.996878 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73,"thread_start_us":44,"threads_started":1}
W20240507 02:07:32.996997 6536 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111) [suppressed 2 similar messages]
I20240507 02:07:32.997298 7282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:32.997218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.0.0.1:39416)
0507 02:07:32.997258 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:32.997282 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20240507 02:07:32.997360 6536 leader_election.cc:334] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 42c4002f2beb495880b7c5db3cabab77 (127.31.131.126:40295): Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
I20240507 02:07:32.997406 6536 leader_election.cc:302] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 063c163f1a014ed18bce90c55921d3ec; no voters: 42c4002f2beb495880b7c5db3cabab77
I20240507 02:07:32.997473 7280 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 02:07:33.074921 7284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:33.074736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.65:58765)
0507 02:07:33.074851 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:33.074878 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":58,"threads_started":1}
I20240507 02:07:33.075026 7285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:33.074857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.67:56575)
0507 02:07:33.074955 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:33.074982 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":50,"threads_started":1}
I20240507 02:07:33.075026 7286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:33.074857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.66:54375)
0507 02:07:33.074959 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:33.074982 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":53,"threads_started":1}
I20240507 02:07:34.075554 7289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:34.075373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.67:56577)
0507 02:07:34.075485 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:34.075513 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":52,"threads_started":1}
I20240507 02:07:34.075554 7287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:34.075332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.65:42577)
0507 02:07:34.075452 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:34.075488 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":61,"threads_started":1}
I20240507 02:07:34.075662 7288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:34.075373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.66:38133)
0507 02:07:34.075475 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:34.075506 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":53,"threads_started":1}
I20240507 02:07:35.076136 7290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:35.075905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.65:36591)
0507 02:07:35.076042 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:35.076078 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":64,"threads_started":1}
I20240507 02:07:35.076246 7291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:35.076037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.66:43837)
0507 02:07:35.076169 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:35.076196 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":53,"threads_started":1}
I20240507 02:07:35.076329 7292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:35.076171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.67:47699)
0507 02:07:35.076266 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:35.076290 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80,"thread_start_us":44,"threads_started":1}
I20240507 02:07:36.076813 7293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:36.076553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.66:33605)
0507 02:07:36.076716 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:36.076753 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":62,"threads_started":1}
I20240507 02:07:36.076872 7294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:36.076539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.65:41009)
0507 02:07:36.076783 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:36.076823 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":63,"threads_started":1}
I20240507 02:07:36.076912 7295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:36.076745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.67:54323)
0507 02:07:36.076847 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:36.076874 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":50,"threads_started":1}
I20240507 02:07:37.077513 7296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:37.077297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.66:50537)
0507 02:07:37.077417 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:37.077453 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":73,"threads_started":1}
I20240507 02:07:37.077584 7298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:37.077353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.65:40429)
0507 02:07:37.077502 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:37.077533 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":55,"threads_started":1}
I20240507 02:07:37.077587 7297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:37.077353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.67:53949)
0507 02:07:37.077524 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:37.077549 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":57,"threads_started":1}
I20240507 02:07:38.078248 7299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:38.077929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.67:50785)
0507 02:07:38.078149 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:38.078188 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":65,"threads_started":1}
I20240507 02:07:38.078327 7300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:38.078095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.66:59945)
0507 02:07:38.078259 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:38.078289 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":66,"threads_started":1}
I20240507 02:07:38.078392 7301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:38.078149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.65:36185)
0507 02:07:38.078247 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:38.078282 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80,"thread_start_us":43,"threads_started":1}
I20240507 02:07:39.079107 7304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:39.078948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.67:59357)
0507 02:07:39.079045 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:39.079069 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":43,"threads_started":1}
I20240507 02:07:39.079095 7302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:39.078768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.66:40323)
0507 02:07:39.078993 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:39.079029 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":82,"threads_started":1}
I20240507 02:07:39.079111 7303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:39.078824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.65:60113)
0507 02:07:39.079026 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:39.079062 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":57,"threads_started":1}
I20240507 02:07:40.079803 7305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:40.079503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.66:56843)
0507 02:07:40.079705 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:40.079743 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":67,"threads_started":1}
I20240507 02:07:40.079849 7307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:40.079534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.65:59401)
0507 02:07:40.079762 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:40.079798 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":52,"threads_started":1}
I20240507 02:07:40.079895 7306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:40.079501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.67:51979)
0507 02:07:40.079826 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:40.079855 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":65,"threads_started":1}
I20240507 02:07:40.383981 7308 raft_consensus.cc:482] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 42c4002f2beb495880b7c5db3cabab77)
I20240507 02:07:40.384089 7308 raft_consensus.cc:504] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "42c4002f2beb495880b7c5db3cabab77" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 40295 } } peers { permanent_uuid: "063c163f1a014ed18bce90c55921d3ec" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 46259 } attrs { promote: false } } peers { permanent_uuid: "65ae2553e5234a48bb69f75f1afe1ea5" member_type: NON_VOTER last_known_addr { host: "127.31.131.124" port: 42683 } attrs { promote: true } }
I20240507 02:07:40.384272 7308 leader_election.cc:288] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42c4002f2beb495880b7c5db3cabab77 (127.31.131.126:40295)
I20240507 02:07:40.384589 7309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:40.384406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.0.0.1:39860)
0507 02:07:40.384505 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:40.384548 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":50,"threads_started":1}
W20240507 02:07:40.384670 6536 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
I20240507 02:07:40.385124 7309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:40.385018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.0.0.1:39874)
0507 02:07:40.385077 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:40.385103 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20240507 02:07:40.385191 6536 leader_election.cc:334] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 42c4002f2beb495880b7c5db3cabab77 (127.31.131.126:40295): Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
I20240507 02:07:40.385228 6536 leader_election.cc:302] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 063c163f1a014ed18bce90c55921d3ec; no voters: 42c4002f2beb495880b7c5db3cabab77
I20240507 02:07:40.385290 7308 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 02:07:41.080672 7312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:41.080385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.65:48243)
0507 02:07:41.080565 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:41.080609 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":56,"threads_started":1}
I20240507 02:07:41.080747 7313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:41.080576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.66:37679)
0507 02:07:41.080671 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:41.080701 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78,"thread_start_us":43,"threads_started":1}
I20240507 02:07:41.080806 7311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:41.080342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.67:42483)
0507 02:07:41.080696 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:41.080722 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":72,"threads_started":1}
I20240507 02:07:42.081404 7314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:42.081073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.65:56375)
0507 02:07:42.081308 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:42.081349 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":63,"threads_started":1}
I20240507 02:07:42.081404 7316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:42.081216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.67:51089)
0507 02:07:42.081312 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:42.081345 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":50,"threads_started":1}
I20240507 02:07:42.081496 7315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:42.081073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.66:39351)
0507 02:07:42.081428 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:42.081458 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":78,"threads_started":1}
I20240507 02:07:43.081967 7318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:43.081748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.67:52111)
0507 02:07:43.081869 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:43.081909 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":70,"threads_started":1}
I20240507 02:07:43.082015 7317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:43.081748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.65:53473)
0507 02:07:43.081929 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:43.081965 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":67,"threads_started":1}
I20240507 02:07:43.082046 7319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:43.081890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.66:40397)
0507 02:07:43.081979 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:43.082007 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76,"thread_start_us":56,"threads_started":1}
I20240507 02:07:44.082804 7320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:44.082397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.65:54227)
0507 02:07:44.082707 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:44.082747 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":82,"threads_started":1}
I20240507 02:07:44.082804 7322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:44.082583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.66:33719)
0507 02:07:44.082695 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:44.082748 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":53,"threads_started":1}
I20240507 02:07:44.083055 7321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:44.082509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.67:51737)
0507 02:07:44.082988 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:44.083019 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":201,"threads_started":1}
I20240507 02:07:45.083505 7323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:45.083255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.66:59947)
0507 02:07:45.083411 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:45.083448 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":64,"threads_started":1}
I20240507 02:07:45.083734 7325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:45.083386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.65:57251)
0507 02:07:45.083645 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:45.083682 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":60,"threads_started":1}
I20240507 02:07:45.083765 7324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:45.083382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.67:46007)
0507 02:07:45.083688 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:45.083720 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":45,"threads_started":1}
I20240507 02:07:46.084444 7326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:46.084216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.66:55653)
0507 02:07:46.084337 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:46.084382 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":70,"threads_started":1}
I20240507 02:07:46.084496 7327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:46.084220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.67:44027)
0507 02:07:46.084428 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:46.084458 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":65,"threads_started":1}
I20240507 02:07:46.084661 7328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:46.084334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.65:51847)
0507 02:07:46.084543 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:46.084608 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":37,"threads_started":1}
I20240507 02:07:47.085214 7329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:47.084908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.67:55305)
0507 02:07:47.085098 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:47.085144 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":68,"threads_started":1}
I20240507 02:07:47.085305 7330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:47.085080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.66:34945)
0507 02:07:47.085231 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:47.085259 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":43,"threads_started":1}
I20240507 02:07:47.085305 7331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:47.085128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.65:58575)
0507 02:07:47.085232 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:47.085259 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":53,"threads_started":1}
I20240507 02:07:48.085935 7333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:48.085729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.65:34125)
0507 02:07:48.085839 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:48.085875 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":53,"threads_started":1}
I20240507 02:07:48.086042 7332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:48.085716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.67:56001)
0507 02:07:48.085972 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:48.086001 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":65,"threads_started":1}
I20240507 02:07:48.086181 7334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:48.085869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.66:42401)
0507 02:07:48.086075 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:48.086109 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":41,"threads_started":1}
I20240507 02:07:48.458914 7335 raft_consensus.cc:482] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 42c4002f2beb495880b7c5db3cabab77)
I20240507 02:07:48.459019 7335 raft_consensus.cc:504] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "42c4002f2beb495880b7c5db3cabab77" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 40295 } } peers { permanent_uuid: "063c163f1a014ed18bce90c55921d3ec" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 46259 } attrs { promote: false } } peers { permanent_uuid: "65ae2553e5234a48bb69f75f1afe1ea5" member_type: NON_VOTER last_known_addr { host: "127.31.131.124" port: 42683 } attrs { promote: true } }
I20240507 02:07:48.459172 7335 leader_election.cc:288] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42c4002f2beb495880b7c5db3cabab77 (127.31.131.126:40295)
I20240507 02:07:48.459483 7336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:48.459317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.0.0.1:43900)
0507 02:07:48.459415 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:48.459444 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":50,"threads_started":1}
W20240507 02:07:48.459594 6536 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
I20240507 02:07:48.459930 7336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:48.459853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.0.0.1:43904)
0507 02:07:48.459892 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:48.459915 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20240507 02:07:48.459992 6536 leader_election.cc:334] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 42c4002f2beb495880b7c5db3cabab77 (127.31.131.126:40295): Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
I20240507 02:07:48.460037 6536 leader_election.cc:302] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 063c163f1a014ed18bce90c55921d3ec; no voters: 42c4002f2beb495880b7c5db3cabab77
I20240507 02:07:48.460108 7335 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 063c163f1a014ed18bce90c55921d3ec [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 02:07:49.086741 7338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:49.086414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.67:48111)
0507 02:07:49.086660 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:49.086689 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":64,"threads_started":1}
I20240507 02:07:49.086742 7340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:49.086515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.66:55223)
0507 02:07:49.086646 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:49.086687 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":73,"threads_started":1}
I20240507 02:07:49.086741 7339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:49.086415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.65:58851)
0507 02:07:49.086654 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:49.086689 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":64,"threads_started":1}
I20240507 02:07:50.087510 7341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:50.087199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.66:54805)
0507 02:07:50.087412 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:50.087451 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":72,"threads_started":1}
I20240507 02:07:50.087666 7342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:50.087380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.67:34603)
0507 02:07:50.087544 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:50.087577 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":55,"threads_started":1}
I20240507 02:07:50.087911 7343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:50.087701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.65:44429)
0507 02:07:50.087799 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:50.087829 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":46,"threads_started":1}
I20240507 02:07:51.088163 7344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:51.087947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.66:54139)
0507 02:07:51.088063 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:51.088101 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":60,"threads_started":1}
I20240507 02:07:51.088322 7345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:51.088142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.67:35141)
0507 02:07:51.088243 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:51.088273 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":52,"threads_started":1}
I20240507 02:07:51.088476 7346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:51.088244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.65:37109)
0507 02:07:51.088408 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:51.088435 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":47,"threads_started":1}
I20240507 02:07:52.089022 7347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:52.088722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.66:60351)
0507 02:07:52.088951 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:52.088985 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":57,"threads_started":1}
I20240507 02:07:52.089022 7348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:52.088708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.67:40449)
0507 02:07:52.088918 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:52.088963 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":65,"threads_started":1}
I20240507 02:07:52.089049 7349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:52.088883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.65:56803)
0507 02:07:52.088980 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:52.089009 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":61,"threads_started":1}
I20240507 02:07:53.089653 7351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:53.089470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.66:53699)
0507 02:07:53.089567 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:53.089596 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":40,"threads_started":1}
I20240507 02:07:53.089653 7350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:53.089414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.67:60415)
0507 02:07:53.089561 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:53.089596 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":80,"threads_started":1}
I20240507 02:07:53.090036 7352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:53.089817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.65:38201)
0507 02:07:53.089939 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:53.089995 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":54,"threads_started":1}
I20240507 02:07:54.090375 7353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:54.090081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.66:43989)
0507 02:07:54.090275 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:54.090316 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":87,"threads_started":1}
I20240507 02:07:54.090690 7354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:54.090106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.67:57333)
0507 02:07:54.090595 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:54.090628 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":103,"threads_started":1}
I20240507 02:07:54.090687 7355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:54.090428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:40295 (local address 127.31.131.65:55329)
0507 02:07:54.090538 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:54.090603 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:40295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":46,"threads_started":1}
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/security-itest.cc:903: Failure
Failed
Bad status: Timed out: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.31.131.126:40295,127.31.131.125:46259,127.31.131.124:42683, num_attempts: 243) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.31.131.126:40295,127.31.131.125:46259,127.31.131.124:42683, num_attempts: 1)
I20240507 02:07:54.220902 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 6697
I20240507 02:07:54.226188 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 6923
I20240507 02:07:54.230250 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 7060
I20240507 02:07:54.234239 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 6416
I20240507 02:07:54.238088 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 6604
2024-05-07T02:07:54Z chronyd exiting
I20240507 02:07:54.253239 32269 test_util.cc:175] -----------------------------------------------
I20240507 02:07:54.253324 32269 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1715047604443657-32269-0
[ FAILED ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (32666 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 07 02:07:54 dist-test-slave-v0bg krb5kdc[7358](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 07 02:07:54 dist-test-slave-v0bg krb5kdc[7358](info): set up 2 sockets
May 07 02:07:54 dist-test-slave-v0bg krb5kdc[7358](info): commencing operation
krb5kdc: starting...
W20240507 02:07:55.123435 32269 mini_kdc.cc:121] Time spent starting KDC: real 0.869s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/krb5kdc/test-user.keytab.
May 07 02:07:55 dist-test-slave-v0bg krb5kdc[7358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-07T02:07:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-07T02:07:55Z Disabled control of system clock
WARNING: no policy specified for oryx/127.31.131.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.131.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.131.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/krb5kdc/oryx_127.31.131.126.keytab.
Entry for principal oryx/127.31.131.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/krb5kdc/oryx_127.31.131.126.keytab.
I20240507 02:07:55.227447 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:43871
--webserver_interface=127.31.131.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/krb5kdc/oryx_127.31.131.126.keytab
--principal=oryx/127.31.131.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:37109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.131.126:43871
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:07:55.304183 7371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:07:55.304358 7371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:07:55.304388 7371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:07:55.305655 7371 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240507 02:07:55.305704 7371 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20240507 02:07:55.305722 7371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:07:55.305737 7371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:07:55.305753 7371 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240507 02:07:55.305768 7371 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240507 02:07:55.307360 7371 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:37109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.131.126:43871
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/krb5kdc/oryx_127.31.131.126.keytab
--principal=oryx/127.31.131.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:43871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.131.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7371
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:07:55.307582 7371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:07:55.307758 7371 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:07:55.311800 7376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:07:55.311869 7377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:07:55.311815 7371 server_base.cc:1034] running on GCE node
W20240507 02:07:55.312041 7379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:07:55.312287 7371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:07:55.312544 7371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:07:55.313679 7371 hybrid_clock.cc:648] HybridClock initialized: now 1715047675313666 us; error 27 us; skew 500 ppm
May 07 02:07:55 dist-test-slave-v0bg krb5kdc[7358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047675, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.131.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:07:55.315950 7371 init.cc:377] Logged in from keytab as oryx/127.31.131.126@KRBTEST.COM (short username oryx)
I20240507 02:07:55.316331 7371 webserver.cc:460] Webserver started at http://127.31.131.126:36933/ using document root <none> and password file <none>
I20240507 02:07:55.316501 7371 fs_manager.cc:359] Metadata directory not provided
I20240507 02:07:55.316541 7371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:07:55.316640 7371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:07:55.317608 7371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/master-0/data/instance:
uuid: "5bf6e9ec74044a588b71bc5772a5d06d"
format_stamp: "Formatted at 2024-05-07 02:07:55 on dist-test-slave-v0bg"
server_key: "4d345c7a7d18a4e18bd753221979c4b4"
server_key_iv: "7652d645706dd5dc253981678ba26ed6"
server_key_version: "encryptionkey@0"
I20240507 02:07:55.317909 7371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/master-0/wal/instance:
uuid: "5bf6e9ec74044a588b71bc5772a5d06d"
format_stamp: "Formatted at 2024-05-07 02:07:55 on dist-test-slave-v0bg"
server_key: "4d345c7a7d18a4e18bd753221979c4b4"
server_key_iv: "7652d645706dd5dc253981678ba26ed6"
server_key_version: "encryptionkey@0"
I20240507 02:07:55.319046 7371 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240507 02:07:55.319634 7386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:55.319784 7371 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240507 02:07:55.319859 7371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/master-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/master-0/wal
uuid: "5bf6e9ec74044a588b71bc5772a5d06d"
format_stamp: "Formatted at 2024-05-07 02:07:55 on dist-test-slave-v0bg"
server_key: "4d345c7a7d18a4e18bd753221979c4b4"
server_key_iv: "7652d645706dd5dc253981678ba26ed6"
server_key_version: "encryptionkey@0"
I20240507 02:07:55.319914 7371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:07:55.331812 7371 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:07:55.333752 7371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:07:55.333866 7371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:07:55.337527 7371 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.126:43871
I20240507 02:07:55.337680 7449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.126:43871 every 8 connection(s)
I20240507 02:07:55.337870 7371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
I20240507 02:07:55.338428 7450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:07:55.340435 7450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bf6e9ec74044a588b71bc5772a5d06d: Bootstrap starting.
I20240507 02:07:55.340948 7450 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5bf6e9ec74044a588b71bc5772a5d06d: Neither blocks nor log segments found. Creating new log.
I20240507 02:07:55.341104 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 7371
I20240507 02:07:55.341194 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/master-0/wal/instance
I20240507 02:07:55.341181 7450 log.cc:824] T 00000000000000000000000000000000 P 5bf6e9ec74044a588b71bc5772a5d06d: Log is configured to *not* fsync() on all Append() calls
I20240507 02:07:55.341370 32269 external_mini_cluster.cc:1428] Setting key 671e765057328ecba1fd79083353ee9e
I20240507 02:07:55.341867 7450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bf6e9ec74044a588b71bc5772a5d06d: No bootstrap required, opened a new log
I20240507 02:07:55.343459 7450 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5bf6e9ec74044a588b71bc5772a5d06d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bf6e9ec74044a588b71bc5772a5d06d" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 43871 } }
I20240507 02:07:55.343578 7450 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5bf6e9ec74044a588b71bc5772a5d06d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:07:55.343593 7450 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5bf6e9ec74044a588b71bc5772a5d06d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bf6e9ec74044a588b71bc5772a5d06d, State: Initialized, Role: FOLLOWER
I20240507 02:07:55.343677 7450 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5bf6e9ec74044a588b71bc5772a5d06d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bf6e9ec74044a588b71bc5772a5d06d" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 43871 } }
I20240507 02:07:55.343739 7450 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5bf6e9ec74044a588b71bc5772a5d06d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 02:07:55.343765 7450 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5bf6e9ec74044a588b71bc5772a5d06d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 02:07:55.343808 7450 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5bf6e9ec74044a588b71bc5772a5d06d [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:07:55.344321 7450 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5bf6e9ec74044a588b71bc5772a5d06d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bf6e9ec74044a588b71bc5772a5d06d" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 43871 } }
I20240507 02:07:55.344406 7450 leader_election.cc:302] T 00000000000000000000000000000000 P 5bf6e9ec74044a588b71bc5772a5d06d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5bf6e9ec74044a588b71bc5772a5d06d; no voters:
I20240507 02:07:55.344537 7450 leader_election.cc:288] T 00000000000000000000000000000000 P 5bf6e9ec74044a588b71bc5772a5d06d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240507 02:07:55.344589 7455 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5bf6e9ec74044a588b71bc5772a5d06d [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:07:55.344760 7450 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5bf6e9ec74044a588b71bc5772a5d06d [sys.catalog]: configured and running, proceeding with master startup.
I20240507 02:07:55.344774 7455 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5bf6e9ec74044a588b71bc5772a5d06d [term 1 LEADER]: Becoming Leader. State: Replica: 5bf6e9ec74044a588b71bc5772a5d06d, State: Running, Role: LEADER
I20240507 02:07:55.344997 7455 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5bf6e9ec74044a588b71bc5772a5d06d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bf6e9ec74044a588b71bc5772a5d06d" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 43871 } }
I20240507 02:07:55.345384 7456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bf6e9ec74044a588b71bc5772a5d06d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5bf6e9ec74044a588b71bc5772a5d06d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bf6e9ec74044a588b71bc5772a5d06d" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 43871 } } }
I20240507 02:07:55.345459 7456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bf6e9ec74044a588b71bc5772a5d06d [sys.catalog]: This master's current role is: LEADER
I20240507 02:07:55.345656 7457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bf6e9ec74044a588b71bc5772a5d06d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5bf6e9ec74044a588b71bc5772a5d06d. Latest consensus state: current_term: 1 leader_uuid: "5bf6e9ec74044a588b71bc5772a5d06d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bf6e9ec74044a588b71bc5772a5d06d" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 43871 } } }
I20240507 02:07:55.345933 7464 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 02:07:55.346412 7464 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240507 02:07:55.346983 7457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bf6e9ec74044a588b71bc5772a5d06d [sys.catalog]: This master's current role is: LEADER
I20240507 02:07:55.347868 7464 catalog_manager.cc:1347] Generated new cluster ID: a6b942a0cf564cc2a7bb997adfeb70e3
I20240507 02:07:55.347910 7464 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
May 07 02:07:55 dist-test-slave-v0bg krb5kdc[7358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM
I20240507 02:07:55.351310 7454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:55.341857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46388 (local address 127.31.131.126:43871)
0507 02:07:55.342047 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:55.342058 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:55.342075 (+ 17us) server_negotiation.cc:407] Connection header received
0507 02:07:55.342898 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:55.342904 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:55.342950 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:55.343073 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:55.343853 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:55.345097 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:55.345918 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:55.346128 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:55.348561 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:55.348582 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:55.348582 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:07:55.348595 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:55.350047 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:55.350331 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:55.350334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:55.350334 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:55.350379 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:55.350503 (+ 124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:55.350504 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:55.350504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:55.350751 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:55.350887 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:55.350997 (+ 110us) server_negotiation.cc:299] Negotiation successful
0507 02:07:55.351110 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":76,"threads_started":1}
I20240507 02:07:55.377465 7464 catalog_manager.cc:1370] Generated new certificate authority record
I20240507 02:07:55.378042 7464 catalog_manager.cc:1504] Loading token signing keys...
I20240507 02:07:55.381975 7464 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 5bf6e9ec74044a588b71bc5772a5d06d: Generated new TSK 0
I20240507 02:07:55.382151 7464 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.31.131.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.131.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.131.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/krb5kdc/oryx_127.31.131.65.keytab.
Entry for principal oryx/127.31.131.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/krb5kdc/oryx_127.31.131.65.keytab.
I20240507 02:07:55.409350 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.65:0
--local_ip_for_outbound_sockets=127.31.131.65
--webserver_interface=127.31.131.65
--webserver_port=0
--tserver_master_addrs=127.31.131.126:43871
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/krb5kdc/oryx_127.31.131.65.keytab
--principal=oryx/127.31.131.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:37109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:07:55.508543 7476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:07:55.508714 7476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:07:55.508744 7476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:07:55.510164 7476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:07:55.510221 7476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:07:55.510247 7476 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20240507 02:07:55.510267 7476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.65
I20240507 02:07:55.511853 7476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:37109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/krb5kdc/oryx_127.31.131.65.keytab
--principal=oryx/127.31.131.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.131.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.131.126:43871
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.65
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:07:55.512058 7476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:07:55.512236 7476 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:07:55.515959 7481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:07:55.516043 7476 server_base.cc:1034] running on GCE node
W20240507 02:07:55.515952 7484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:07:55.515952 7482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:07:55.516328 7476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:07:55.516530 7476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:07:55.517675 7476 hybrid_clock.cc:648] HybridClock initialized: now 1715047675517660 us; error 31 us; skew 500 ppm
May 07 02:07:55 dist-test-slave-v0bg krb5kdc[7358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047675, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.131.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:07:55.519960 7476 init.cc:377] Logged in from keytab as oryx/127.31.131.65@KRBTEST.COM (short username oryx)
I20240507 02:07:55.520363 7476 webserver.cc:460] Webserver started at http://127.31.131.65:45009/ using document root <none> and password file <none>
I20240507 02:07:55.520539 7476 fs_manager.cc:359] Metadata directory not provided
I20240507 02:07:55.520576 7476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:07:55.520675 7476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:07:55.521620 7476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/ts-0/data/instance:
uuid: "b442a50dc84d4fb0bc141f691a5e8184"
format_stamp: "Formatted at 2024-05-07 02:07:55 on dist-test-slave-v0bg"
server_key: "95850ce098767ea7f0f9bb79473caa66"
server_key_iv: "eac1ba3bd7274bf68ee37cb35e904595"
server_key_version: "encryptionkey@0"
I20240507 02:07:55.521894 7476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/ts-0/wal/instance:
uuid: "b442a50dc84d4fb0bc141f691a5e8184"
format_stamp: "Formatted at 2024-05-07 02:07:55 on dist-test-slave-v0bg"
server_key: "95850ce098767ea7f0f9bb79473caa66"
server_key_iv: "eac1ba3bd7274bf68ee37cb35e904595"
server_key_version: "encryptionkey@0"
I20240507 02:07:55.523039 7476 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240507 02:07:55.523648 7491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:55.523846 7476 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240507 02:07:55.523917 7476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/ts-0/wal
uuid: "b442a50dc84d4fb0bc141f691a5e8184"
format_stamp: "Formatted at 2024-05-07 02:07:55 on dist-test-slave-v0bg"
server_key: "95850ce098767ea7f0f9bb79473caa66"
server_key_iv: "eac1ba3bd7274bf68ee37cb35e904595"
server_key_version: "encryptionkey@0"
I20240507 02:07:55.523980 7476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:07:55.541224 7476 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:07:55.543221 7476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:07:55.543329 7476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:07:55.543862 7476 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:07:55.543901 7476 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:55.543926 7476 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:07:55.543941 7476 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
May 07 02:07:55 dist-test-slave-v0bg krb5kdc[7358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047675, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.131.65@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM
I20240507 02:07:55.550694 7476 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.65:35717
I20240507 02:07:55.550738 7606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.65:35717 every 8 connection(s)
I20240507 02:07:55.551051 7476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
I20240507 02:07:55.551911 7454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:55.544214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:46289 (local address 127.31.131.126:43871)
0507 02:07:55.544262 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:55.544264 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:55.544445 (+ 181us) server_negotiation.cc:407] Connection header received
0507 02:07:55.544952 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:55.544953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:55.544966 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:55.545032 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:55.545764 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:55.546527 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:55.547106 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:55.547287 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:55.549220 (+ 1933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:55.549225 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:55.549225 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:07:55.549232 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:55.550840 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:55.551163 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:55.551164 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:55.551164 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:55.551190 (+ 26us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:55.551378 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:55.551380 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:55.551380 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:55.551486 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:55.551541 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:55.551764 (+ 223us) server_negotiation.cc:299] Negotiation successful
0507 02:07:55.551847 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20240507 02:07:55.551980 7500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:55.544214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:43871 (local address 127.31.131.65:46289)
0507 02:07:55.544349 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:55.544372 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:55.544854 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:55.545087 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:55.545090 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:55.545393 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:07:55.545674 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:55.545678 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:55.546647 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:55.546648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:55.547043 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:55.547045 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:55.547162 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:55.547653 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:07:55.547672 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:07:55.549123 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:07:55.550912 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:55.550915 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:55.550917 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:55.551093 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:55.551243 (+ 150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:55.551244 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:55.551244 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:55.551311 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:55.551539 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:07:55.551543 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:07:55.551650 (+ 107us) client_negotiation.cc:769] Sending connection context
0507 02:07:55.551748 (+ 98us) client_negotiation.cc:240] Negotiation successful
0507 02:07:55.551850 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":79,"threads_started":1}
I20240507 02:07:55.552419 7607 heartbeater.cc:347] Connected to a master server at 127.31.131.126:43871
I20240507 02:07:55.552498 7607 heartbeater.cc:464] Registering TS with master...
I20240507 02:07:55.552703 7607 heartbeater.cc:511] Master 127.31.131.126:43871 requested a full tablet report, sending...
I20240507 02:07:55.553228 7402 ts_manager.cc:194] Registered new tserver with Master: b442a50dc84d4fb0bc141f691a5e8184 (127.31.131.65:35717)
I20240507 02:07:55.553994 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 7476
I20240507 02:07:55.554019 7402 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.131.65@KRBTEST.COM'} at 127.31.131.65:46289
I20240507 02:07:55.554092 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/ts-0/wal/instance
I20240507 02:07:55.554271 32269 external_mini_cluster.cc:1428] Setting key bfaf26cab25c548ddad391536d16804c
I20240507 02:07:55.558180 7454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:55.553393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:49029 (local address 127.31.131.126:43871)
0507 02:07:55.553429 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:55.553431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:55.553561 (+ 130us) server_negotiation.cc:407] Connection header received
0507 02:07:55.553601 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:55.553602 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:55.553612 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:55.553658 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:55.554152 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:55.554781 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:55.555411 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:55.555527 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:55.556137 (+ 610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:55.556140 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:55.556140 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:07:55.556144 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:55.557377 (+ 1233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:55.557629 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:55.557630 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:55.557631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:55.557649 (+ 18us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:55.557771 (+ 122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:55.557772 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:55.557772 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:55.557851 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:55.557902 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:55.558026 (+ 124us) server_negotiation.cc:299] Negotiation successful
0507 02:07:55.558091 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20240507 02:07:55.558199 7613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:55.553358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:43871 (local address 127.31.131.65:49029)
0507 02:07:55.553489 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:55.553499 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:55.553554 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:55.553678 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:55.553679 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:55.553792 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:07:55.554054 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:55.554056 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:55.554851 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:55.554852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:55.555338 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:55.555341 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:55.555407 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:55.555787 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:07:55.555791 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:07:55.556067 (+ 276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:07:55.557446 (+ 1379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:55.557447 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:55.557447 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:55.557575 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:55.557696 (+ 121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:55.557696 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:55.557697 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:55.557724 (+ 27us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:55.557900 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:07:55.557901 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:07:55.557939 (+ 38us) client_negotiation.cc:769] Sending connection context
0507 02:07:55.557998 (+ 59us) client_negotiation.cc:240] Negotiation successful
0507 02:07:55.558091 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":72,"threads_started":1}
WARNING: no policy specified for oryx/127.31.131.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.131.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.131.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/krb5kdc/oryx_127.31.131.66.keytab.
Entry for principal oryx/127.31.131.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/krb5kdc/oryx_127.31.131.66.keytab.
I20240507 02:07:55.577685 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.66:0
--local_ip_for_outbound_sockets=127.31.131.66
--webserver_interface=127.31.131.66
--webserver_port=0
--tserver_master_addrs=127.31.131.126:43871
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/krb5kdc/oryx_127.31.131.66.keytab
--principal=oryx/127.31.131.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:37109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:07:55.654209 7616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:07:55.654395 7616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:07:55.654420 7616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:07:55.655726 7616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:07:55.655778 7616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:07:55.655800 7616 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20240507 02:07:55.655833 7616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.66
I20240507 02:07:55.657426 7616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:37109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/krb5kdc/oryx_127.31.131.66.keytab
--principal=oryx/127.31.131.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.131.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.131.126:43871
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.66
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:07:55.657630 7616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:07:55.657815 7616 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:07:55.661718 7625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:07:55.661777 7616 server_base.cc:1034] running on GCE node
W20240507 02:07:55.661883 7623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:07:55.661986 7622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:07:55.662163 7616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:07:55.662380 7616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:07:55.663507 7616 hybrid_clock.cc:648] HybridClock initialized: now 1715047675663484 us; error 38 us; skew 500 ppm
May 07 02:07:55 dist-test-slave-v0bg krb5kdc[7358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047675, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.131.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:07:55.665761 7616 init.cc:377] Logged in from keytab as oryx/127.31.131.66@KRBTEST.COM (short username oryx)
I20240507 02:07:55.666174 7616 webserver.cc:460] Webserver started at http://127.31.131.66:43983/ using document root <none> and password file <none>
I20240507 02:07:55.666342 7616 fs_manager.cc:359] Metadata directory not provided
I20240507 02:07:55.666380 7616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:07:55.666481 7616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:07:55.667456 7616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/ts-1/data/instance:
uuid: "11db10e937ef426c9344e815d3e7262c"
format_stamp: "Formatted at 2024-05-07 02:07:55 on dist-test-slave-v0bg"
server_key: "9607f57b135faf815a3ac9dd034887d9"
server_key_iv: "a1fab623ea02687777a64678732e0b5d"
server_key_version: "encryptionkey@0"
I20240507 02:07:55.667735 7616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/ts-1/wal/instance:
uuid: "11db10e937ef426c9344e815d3e7262c"
format_stamp: "Formatted at 2024-05-07 02:07:55 on dist-test-slave-v0bg"
server_key: "9607f57b135faf815a3ac9dd034887d9"
server_key_iv: "a1fab623ea02687777a64678732e0b5d"
server_key_version: "encryptionkey@0"
I20240507 02:07:55.668798 7616 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240507 02:07:55.669371 7632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:55.669523 7616 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240507 02:07:55.669592 7616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/ts-1/wal
uuid: "11db10e937ef426c9344e815d3e7262c"
format_stamp: "Formatted at 2024-05-07 02:07:55 on dist-test-slave-v0bg"
server_key: "9607f57b135faf815a3ac9dd034887d9"
server_key_iv: "a1fab623ea02687777a64678732e0b5d"
server_key_version: "encryptionkey@0"
I20240507 02:07:55.669646 7616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:07:55.683651 7616 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:07:55.685619 7616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:07:55.685727 7616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:07:55.686264 7616 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:07:55.686308 7616 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:55.686353 7616 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:07:55.686378 7616 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
May 07 02:07:55 dist-test-slave-v0bg krb5kdc[7358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047675, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.131.66@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM
I20240507 02:07:55.692839 7616 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.66:44619
I20240507 02:07:55.693149 7616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
I20240507 02:07:55.693537 7749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.66:44619 every 8 connection(s)
I20240507 02:07:55.693956 7454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:55.686601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.66:55969 (local address 127.31.131.126:43871)
0507 02:07:55.686669 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:55.686672 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:55.686851 (+ 179us) server_negotiation.cc:407] Connection header received
0507 02:07:55.687360 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:55.687362 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:55.687375 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:55.687452 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:55.688226 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:55.688711 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:55.689216 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:55.689380 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:55.691335 (+ 1955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:55.691340 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:55.691340 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:07:55.691347 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:55.692770 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:55.693133 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:55.693134 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:55.693134 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:55.693158 (+ 24us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:55.693327 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:55.693328 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:55.693328 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:55.693443 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:55.693502 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:55.693793 (+ 291us) server_negotiation.cc:299] Negotiation successful
0507 02:07:55.693878 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20240507 02:07:55.694015 7641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:55.686584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:43871 (local address 127.31.131.66:55969)
0507 02:07:55.686743 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:55.686772 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:55.687229 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:55.687525 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:55.687529 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:55.687839 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:07:55.688128 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:55.688132 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:55.688789 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:55.688790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:55.689146 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:55.689147 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:55.689268 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:55.689802 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:07:55.689820 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:07:55.691234 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:07:55.692849 (+ 1615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:55.692853 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:55.692855 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:55.693071 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:55.693207 (+ 136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:55.693208 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:55.693209 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:55.693282 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:55.693527 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:07:55.693531 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:07:55.693656 (+ 125us) client_negotiation.cc:769] Sending connection context
0507 02:07:55.693754 (+ 98us) client_negotiation.cc:240] Negotiation successful
0507 02:07:55.693858 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":80,"threads_started":1}
I20240507 02:07:55.694413 7750 heartbeater.cc:347] Connected to a master server at 127.31.131.126:43871
I20240507 02:07:55.694489 7750 heartbeater.cc:464] Registering TS with master...
I20240507 02:07:55.694694 7750 heartbeater.cc:511] Master 127.31.131.126:43871 requested a full tablet report, sending...
I20240507 02:07:55.695106 7403 ts_manager.cc:194] Registered new tserver with Master: 11db10e937ef426c9344e815d3e7262c (127.31.131.66:44619)
I20240507 02:07:55.695590 7403 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.131.66@KRBTEST.COM'} at 127.31.131.66:55969
I20240507 02:07:55.699482 7454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:55.695407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.66:47893 (local address 127.31.131.126:43871)
0507 02:07:55.695445 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:55.695447 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:55.695553 (+ 106us) server_negotiation.cc:407] Connection header received
0507 02:07:55.695604 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:55.695605 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:55.695613 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:55.695657 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:55.696096 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:55.696537 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:55.697011 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:55.697108 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:55.697670 (+ 562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:55.697673 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:55.697673 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:07:55.697678 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:55.698836 (+ 1158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:55.699043 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:55.699044 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:55.699044 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:55.699059 (+ 15us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:55.699148 (+ 89us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:55.699148 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:55.699149 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:55.699238 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:55.699279 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:55.699367 (+ 88us) server_negotiation.cc:299] Negotiation successful
0507 02:07:55.699425 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20240507 02:07:55.699494 7756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:55.695338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:43871 (local address 127.31.131.66:47893)
0507 02:07:55.695485 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:55.695495 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:55.695555 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:55.695662 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:55.695662 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:55.695793 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:07:55.696028 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:55.696030 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:55.696599 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:55.696600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:55.696954 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:55.696956 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:55.697008 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:55.697400 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:07:55.697403 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:07:55.697609 (+ 206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:07:55.698893 (+ 1284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:55.698893 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:55.698893 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:55.698997 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:55.699099 (+ 102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:55.699100 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:55.699100 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:55.699119 (+ 19us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:55.699278 (+ 159us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:07:55.699279 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:07:55.699310 (+ 31us) client_negotiation.cc:769] Sending connection context
0507 02:07:55.699354 (+ 44us) client_negotiation.cc:240] Negotiation successful
0507 02:07:55.699420 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":62,"threads_started":1}
I20240507 02:07:55.701913 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 7616
I20240507 02:07:55.702008 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/ts-1/wal/instance
I20240507 02:07:55.702162 32269 external_mini_cluster.cc:1428] Setting key bc2ddf51397585ab7010e3f72962adf3
WARNING: no policy specified for oryx/127.31.131.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.131.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.131.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/krb5kdc/oryx_127.31.131.67.keytab.
Entry for principal oryx/127.31.131.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/krb5kdc/oryx_127.31.131.67.keytab.
I20240507 02:07:55.722982 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.67:0
--local_ip_for_outbound_sockets=127.31.131.67
--webserver_interface=127.31.131.67
--webserver_port=0
--tserver_master_addrs=127.31.131.126:43871
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/krb5kdc/oryx_127.31.131.67.keytab
--principal=oryx/127.31.131.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:37109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:07:55.798226 7759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:07:55.798404 7759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:07:55.798426 7759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:07:55.799703 7759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:07:55.799744 7759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:07:55.799762 7759 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20240507 02:07:55.799790 7759 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.67
I20240507 02:07:55.801308 7759 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:37109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/krb5kdc/oryx_127.31.131.67.keytab
--principal=oryx/127.31.131.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.131.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.131.126:43871
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.67
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:07:55.801512 7759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:07:55.801687 7759 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:07:55.805328 7764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:07:55.805338 7767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:07:55.805368 7759 server_base.cc:1034] running on GCE node
W20240507 02:07:55.805495 7765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:07:55.805677 7759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:07:55.805853 7759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:07:55.806982 7759 hybrid_clock.cc:648] HybridClock initialized: now 1715047675806974 us; error 27 us; skew 500 ppm
May 07 02:07:55 dist-test-slave-v0bg krb5kdc[7358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047675, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.131.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:07:55.809159 7759 init.cc:377] Logged in from keytab as oryx/127.31.131.67@KRBTEST.COM (short username oryx)
I20240507 02:07:55.809540 7759 webserver.cc:460] Webserver started at http://127.31.131.67:46427/ using document root <none> and password file <none>
I20240507 02:07:55.809697 7759 fs_manager.cc:359] Metadata directory not provided
I20240507 02:07:55.809734 7759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:07:55.809830 7759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:07:55.810791 7759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/ts-2/data/instance:
uuid: "403faea0cec54c4ba72e8997f14141f2"
format_stamp: "Formatted at 2024-05-07 02:07:55 on dist-test-slave-v0bg"
server_key: "78a0a1b15d1e7b99e6f4c6b8bae9ce2c"
server_key_iv: "f9d9f6b4f0552aacdcb764b11b1189aa"
server_key_version: "encryptionkey@0"
I20240507 02:07:55.811070 7759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/ts-2/wal/instance:
uuid: "403faea0cec54c4ba72e8997f14141f2"
format_stamp: "Formatted at 2024-05-07 02:07:55 on dist-test-slave-v0bg"
server_key: "78a0a1b15d1e7b99e6f4c6b8bae9ce2c"
server_key_iv: "f9d9f6b4f0552aacdcb764b11b1189aa"
server_key_version: "encryptionkey@0"
I20240507 02:07:55.812119 7759 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240507 02:07:55.812752 7774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:55.812899 7759 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240507 02:07:55.812979 7759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/ts-2/wal
uuid: "403faea0cec54c4ba72e8997f14141f2"
format_stamp: "Formatted at 2024-05-07 02:07:55 on dist-test-slave-v0bg"
server_key: "78a0a1b15d1e7b99e6f4c6b8bae9ce2c"
server_key_iv: "f9d9f6b4f0552aacdcb764b11b1189aa"
server_key_version: "encryptionkey@0"
I20240507 02:07:55.813037 7759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:07:55.841188 7759 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:07:55.842804 7759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:07:55.842913 7759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:07:55.843389 7759 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:07:55.843428 7759 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:55.843459 7759 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:07:55.843487 7759 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:55.849058 7759 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.67:36381
I20240507 02:07:55.849112 7890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.67:36381 every 8 connection(s)
May 07 02:07:55 dist-test-slave-v0bg krb5kdc[7358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047675, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.131.67@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM
I20240507 02:07:55.849480 7759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
I20240507 02:07:55.852087 7454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:55.843740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.67:43373 (local address 127.31.131.126:43871)
0507 02:07:55.843800 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:55.843803 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:55.843971 (+ 168us) server_negotiation.cc:407] Connection header received
0507 02:07:55.844542 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:55.844543 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:55.844556 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:55.844601 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:55.845695 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:55.846472 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:55.847102 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:55.847282 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:55.849867 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:55.849871 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:55.849871 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:07:55.849877 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:55.851088 (+ 1211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:55.851480 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:55.851481 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:55.851481 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:55.851496 (+ 15us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:55.851655 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:55.851655 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:55.851656 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:55.851731 (+ 75us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:55.851777 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:55.851971 (+ 194us) server_negotiation.cc:299] Negotiation successful
0507 02:07:55.852025 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20240507 02:07:55.852219 7783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:55.843704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:43871 (local address 127.31.131.67:43373)
0507 02:07:55.843858 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:55.843885 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:55.844423 (+ 538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:55.844734 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:55.844738 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:55.845179 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:07:55.845587 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:55.845593 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:55.846602 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:55.846603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:55.847034 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:55.847037 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:55.847159 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:55.847687 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:07:55.847708 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:07:55.849775 (+ 2067us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:07:55.851166 (+ 1391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:55.851169 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:55.851171 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:55.851411 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:55.851545 (+ 134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:55.851546 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:55.851547 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:55.851613 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:55.851775 (+ 162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:07:55.851777 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:07:55.851876 (+ 99us) client_negotiation.cc:769] Sending connection context
0507 02:07:55.851954 (+ 78us) client_negotiation.cc:240] Negotiation successful
0507 02:07:55.852050 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":96,"threads_started":1}
I20240507 02:07:55.852675 7891 heartbeater.cc:347] Connected to a master server at 127.31.131.126:43871
I20240507 02:07:55.852754 7891 heartbeater.cc:464] Registering TS with master...
I20240507 02:07:55.852932 7891 heartbeater.cc:511] Master 127.31.131.126:43871 requested a full tablet report, sending...
I20240507 02:07:55.853293 7402 ts_manager.cc:194] Registered new tserver with Master: 403faea0cec54c4ba72e8997f14141f2 (127.31.131.67:36381)
I20240507 02:07:55.853719 7402 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.131.67@KRBTEST.COM'} at 127.31.131.67:43373
I20240507 02:07:55.856799 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 7759
I20240507 02:07:55.856873 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/ts-2/wal/instance
I20240507 02:07:55.857019 32269 external_mini_cluster.cc:1428] Setting key 528a8b9b773451b3ccdeec9290c3e406
I20240507 02:07:55.857725 32269 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240507 02:07:55.858018 7454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:55.853644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.67:48809 (local address 127.31.131.126:43871)
0507 02:07:55.853682 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:55.853683 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:55.853757 (+ 74us) server_negotiation.cc:407] Connection header received
0507 02:07:55.853811 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:55.853812 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:55.853821 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:55.853873 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:55.854469 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:55.854935 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:55.855406 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:55.855500 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:55.856014 (+ 514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:55.856017 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:55.856017 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:07:55.856021 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:55.857219 (+ 1198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:55.857469 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:55.857471 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:55.857471 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:55.857492 (+ 21us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:55.857608 (+ 116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:55.857608 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:55.857609 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:55.857694 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:55.857742 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:55.857864 (+ 122us) server_negotiation.cc:299] Negotiation successful
0507 02:07:55.857928 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20240507 02:07:55.858033 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:55.853581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:43871 (local address 127.31.131.67:48809)
0507 02:07:55.853689 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:55.853699 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:55.853767 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:55.853989 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:55.853990 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:55.854166 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:07:55.854401 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:55.854402 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:55.854993 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:55.854993 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:55.855352 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:55.855354 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:55.855405 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:55.855750 (+ 345us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:07:55.855753 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:07:55.855954 (+ 201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:07:55.857293 (+ 1339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:55.857294 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:55.857294 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:55.857417 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:55.857534 (+ 117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:55.857535 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:55.857535 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:55.857562 (+ 27us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:55.857742 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:07:55.857742 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:07:55.857780 (+ 38us) client_negotiation.cc:769] Sending connection context
0507 02:07:55.857841 (+ 61us) client_negotiation.cc:240] Negotiation successful
0507 02:07:55.857928 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":57,"threads_started":1}
May 07 02:07:55 dist-test-slave-v0bg krb5kdc[7358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047675, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
May 07 02:07:55 dist-test-slave-v0bg krb5kdc[7358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
May 07 02:07:55 dist-test-slave-v0bg krb5kdc[7358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
W20240507 02:07:55.875818 7454 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:07:55.871196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46394 (local address 127.31.131.126:43871)
0507 02:07:55.871272 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:55.871274 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:55.871297 (+ 23us) server_negotiation.cc:407] Connection header received
0507 02:07:55.871323 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:55.871323 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:55.871332 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:55.871372 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:55.872006 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:55.872551 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:55.873094 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:55.873254 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:55.875737 (+ 2483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46394: BlockingRecv error: recv got EOF from 127.0.0.1:46394 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
May 07 02:07:55 dist-test-slave-v0bg krb5kdc[7358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047675, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM
I20240507 02:07:55.882289 7454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:55.876566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46408 (local address 127.31.131.126:43871)
0507 02:07:55.876622 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:55.876624 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:55.876660 (+ 36us) server_negotiation.cc:407] Connection header received
0507 02:07:55.876736 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:55.876737 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:55.876747 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:55.876807 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:55.877352 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:55.877806 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:55.878305 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:55.878455 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:55.880208 (+ 1753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:55.880212 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:55.880212 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:07:55.880219 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:55.881308 (+ 1089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:55.881582 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:55.881583 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:55.881584 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:55.881608 (+ 24us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:55.881771 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:55.881771 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:55.881772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:55.881867 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:55.881967 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:55.882102 (+ 135us) server_negotiation.cc:299] Negotiation successful
0507 02:07:55.882190 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20240507 02:07:55.883811 7402 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46408:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 02:07:55.885226 7402 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 02:07:55.889993 7913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:55.887246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.65:35717 (local address 127.0.0.1:51620)
0507 02:07:55.887559 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:55.887579 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:55.887656 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:55.887832 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:55.887835 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:55.887837 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:07:55.888073 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:55.888075 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:55.889165 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:55.889166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:55.889720 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:55.889721 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:55.889786 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:55.889824 (+ 38us) client_negotiation.cc:769] Sending connection context
0507 02:07:55.889871 (+ 47us) client_negotiation.cc:240] Negotiation successful
0507 02:07:55.889908 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":74,"threads_started":1}
I20240507 02:07:55.890499 7917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:55.887226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.66:44619 (local address 127.0.0.1:56860)
0507 02:07:55.887549 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:55.887578 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:55.887667 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:55.887875 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:55.887876 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:55.887878 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:07:55.888079 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:55.888081 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:55.889390 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:55.889391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:55.890249 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:55.890252 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:55.890350 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:55.890354 (+ 4us) client_negotiation.cc:769] Sending connection context
0507 02:07:55.890389 (+ 35us) client_negotiation.cc:240] Negotiation successful
0507 02:07:55.890430 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":46,"threads_started":1}
I20240507 02:07:55.890722 7914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:55.887429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.67:36381 (local address 127.0.0.1:57052)
0507 02:07:55.887977 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:55.887986 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:55.888031 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:55.888325 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:55.888326 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:55.888328 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:07:55.888520 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:55.888521 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:55.889803 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:55.889804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:55.890496 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:55.890499 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:55.890594 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:55.890599 (+ 5us) client_negotiation.cc:769] Sending connection context
0507 02:07:55.890629 (+ 30us) client_negotiation.cc:240] Negotiation successful
0507 02:07:55.890664 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":59,"threads_started":1}
I20240507 02:07:55.891088 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:55.887319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56860 (local address 127.31.131.66:44619)
0507 02:07:55.887563 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:55.887569 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:55.887744 (+ 175us) server_negotiation.cc:407] Connection header received
0507 02:07:55.887781 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:55.887783 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:55.887796 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:55.887868 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:07:55.888185 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:55.889280 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:55.890372 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:55.890923 (+ 551us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:55.890973 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:55.891004 (+ 31us) server_negotiation.cc:299] Negotiation successful
0507 02:07:55.891033 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":94,"threads_started":1}
I20240507 02:07:55.891357 7918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:55.887534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57052 (local address 127.31.131.67:36381)
0507 02:07:55.888142 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:55.888150 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:55.888159 (+ 9us) server_negotiation.cc:407] Connection header received
0507 02:07:55.888197 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:55.888198 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:55.888215 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:55.888328 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:07:55.888594 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:55.889714 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:55.890688 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:55.891180 (+ 492us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:55.891225 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:55.891255 (+ 30us) server_negotiation.cc:299] Negotiation successful
0507 02:07:55.891292 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":94,"threads_started":1}
I20240507 02:07:55.891505 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:55.887369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51620 (local address 127.31.131.65:35717)
0507 02:07:55.887513 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:55.887521 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:55.887651 (+ 130us) server_negotiation.cc:407] Connection header received
0507 02:07:55.887746 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:55.887748 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:55.887761 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:55.887826 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:07:55.888263 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:55.889099 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:55.890839 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:55.891329 (+ 490us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:55.891365 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:55.891394 (+ 29us) server_negotiation.cc:299] Negotiation successful
0507 02:07:55.891435 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":67,"threads_started":1}
I20240507 02:07:55.891794 7683 tablet_service.cc:1467] Processing CreateTablet for tablet 7aa9dc252ae84644ae1084ee397d92de (DEFAULT_TABLE table=test-table [id=c66f119023db4092b1547ed08c66aefa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:07:55.891815 7824 tablet_service.cc:1467] Processing CreateTablet for tablet 7aa9dc252ae84644ae1084ee397d92de (DEFAULT_TABLE table=test-table [id=c66f119023db4092b1547ed08c66aefa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:07:55.891986 7531 tablet_service.cc:1467] Processing CreateTablet for tablet 7aa9dc252ae84644ae1084ee397d92de (DEFAULT_TABLE table=test-table [id=c66f119023db4092b1547ed08c66aefa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:07:55.892089 7683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7aa9dc252ae84644ae1084ee397d92de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:07:55.892089 7824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7aa9dc252ae84644ae1084ee397d92de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:07:55.892185 7531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7aa9dc252ae84644ae1084ee397d92de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:07:55.895102 7920 tablet_bootstrap.cc:492] T 7aa9dc252ae84644ae1084ee397d92de P 403faea0cec54c4ba72e8997f14141f2: Bootstrap starting.
I20240507 02:07:55.895244 7919 tablet_bootstrap.cc:492] T 7aa9dc252ae84644ae1084ee397d92de P b442a50dc84d4fb0bc141f691a5e8184: Bootstrap starting.
I20240507 02:07:55.895911 7920 tablet_bootstrap.cc:654] T 7aa9dc252ae84644ae1084ee397d92de P 403faea0cec54c4ba72e8997f14141f2: Neither blocks nor log segments found. Creating new log.
I20240507 02:07:55.896057 7919 tablet_bootstrap.cc:654] T 7aa9dc252ae84644ae1084ee397d92de P b442a50dc84d4fb0bc141f691a5e8184: Neither blocks nor log segments found. Creating new log.
I20240507 02:07:55.896152 7920 log.cc:824] T 7aa9dc252ae84644ae1084ee397d92de P 403faea0cec54c4ba72e8997f14141f2: Log is configured to *not* fsync() on all Append() calls
I20240507 02:07:55.896327 7919 log.cc:824] T 7aa9dc252ae84644ae1084ee397d92de P b442a50dc84d4fb0bc141f691a5e8184: Log is configured to *not* fsync() on all Append() calls
I20240507 02:07:55.896417 7921 tablet_bootstrap.cc:492] T 7aa9dc252ae84644ae1084ee397d92de P 11db10e937ef426c9344e815d3e7262c: Bootstrap starting.
I20240507 02:07:55.896879 7920 tablet_bootstrap.cc:492] T 7aa9dc252ae84644ae1084ee397d92de P 403faea0cec54c4ba72e8997f14141f2: No bootstrap required, opened a new log
I20240507 02:07:55.896945 7920 ts_tablet_manager.cc:1367] T 7aa9dc252ae84644ae1084ee397d92de P 403faea0cec54c4ba72e8997f14141f2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240507 02:07:55.897092 7919 tablet_bootstrap.cc:492] T 7aa9dc252ae84644ae1084ee397d92de P b442a50dc84d4fb0bc141f691a5e8184: No bootstrap required, opened a new log
I20240507 02:07:55.897208 7919 ts_tablet_manager.cc:1367] T 7aa9dc252ae84644ae1084ee397d92de P b442a50dc84d4fb0bc141f691a5e8184: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240507 02:07:55.898598 7920 raft_consensus.cc:348] T 7aa9dc252ae84644ae1084ee397d92de P 403faea0cec54c4ba72e8997f14141f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b442a50dc84d4fb0bc141f691a5e8184" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35717 } } peers { permanent_uuid: "11db10e937ef426c9344e815d3e7262c" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44619 } } peers { permanent_uuid: "403faea0cec54c4ba72e8997f14141f2" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36381 } }
I20240507 02:07:55.898733 7920 raft_consensus.cc:374] T 7aa9dc252ae84644ae1084ee397d92de P 403faea0cec54c4ba72e8997f14141f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:07:55.898766 7920 raft_consensus.cc:729] T 7aa9dc252ae84644ae1084ee397d92de P 403faea0cec54c4ba72e8997f14141f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 403faea0cec54c4ba72e8997f14141f2, State: Initialized, Role: FOLLOWER
I20240507 02:07:55.898869 7920 consensus_queue.cc:260] T 7aa9dc252ae84644ae1084ee397d92de P 403faea0cec54c4ba72e8997f14141f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b442a50dc84d4fb0bc141f691a5e8184" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35717 } } peers { permanent_uuid: "11db10e937ef426c9344e815d3e7262c" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44619 } } peers { permanent_uuid: "403faea0cec54c4ba72e8997f14141f2" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36381 } }
I20240507 02:07:55.899096 7920 ts_tablet_manager.cc:1398] T 7aa9dc252ae84644ae1084ee397d92de P 403faea0cec54c4ba72e8997f14141f2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240507 02:07:55.899179 7919 raft_consensus.cc:348] T 7aa9dc252ae84644ae1084ee397d92de P b442a50dc84d4fb0bc141f691a5e8184 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b442a50dc84d4fb0bc141f691a5e8184" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35717 } } peers { permanent_uuid: "11db10e937ef426c9344e815d3e7262c" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44619 } } peers { permanent_uuid: "403faea0cec54c4ba72e8997f14141f2" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36381 } }
I20240507 02:07:55.899313 7919 raft_consensus.cc:374] T 7aa9dc252ae84644ae1084ee397d92de P b442a50dc84d4fb0bc141f691a5e8184 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:07:55.899339 7919 raft_consensus.cc:729] T 7aa9dc252ae84644ae1084ee397d92de P b442a50dc84d4fb0bc141f691a5e8184 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b442a50dc84d4fb0bc141f691a5e8184, State: Initialized, Role: FOLLOWER
I20240507 02:07:55.899430 7919 consensus_queue.cc:260] T 7aa9dc252ae84644ae1084ee397d92de P b442a50dc84d4fb0bc141f691a5e8184 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b442a50dc84d4fb0bc141f691a5e8184" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35717 } } peers { permanent_uuid: "11db10e937ef426c9344e815d3e7262c" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44619 } } peers { permanent_uuid: "403faea0cec54c4ba72e8997f14141f2" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36381 } }
I20240507 02:07:55.899664 7921 tablet_bootstrap.cc:654] T 7aa9dc252ae84644ae1084ee397d92de P 11db10e937ef426c9344e815d3e7262c: Neither blocks nor log segments found. Creating new log.
I20240507 02:07:55.899667 7919 ts_tablet_manager.cc:1398] T 7aa9dc252ae84644ae1084ee397d92de P b442a50dc84d4fb0bc141f691a5e8184: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240507 02:07:55.899714 7607 heartbeater.cc:503] Master 127.31.131.126:43871 was elected leader, sending a full tablet report...
I20240507 02:07:55.899932 7921 log.cc:824] T 7aa9dc252ae84644ae1084ee397d92de P 11db10e937ef426c9344e815d3e7262c: Log is configured to *not* fsync() on all Append() calls
I20240507 02:07:55.900182 7891 heartbeater.cc:503] Master 127.31.131.126:43871 was elected leader, sending a full tablet report...
I20240507 02:07:55.900732 7921 tablet_bootstrap.cc:492] T 7aa9dc252ae84644ae1084ee397d92de P 11db10e937ef426c9344e815d3e7262c: No bootstrap required, opened a new log
I20240507 02:07:55.900802 7921 ts_tablet_manager.cc:1367] T 7aa9dc252ae84644ae1084ee397d92de P 11db10e937ef426c9344e815d3e7262c: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240507 02:07:55.901971 7921 raft_consensus.cc:348] T 7aa9dc252ae84644ae1084ee397d92de P 11db10e937ef426c9344e815d3e7262c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b442a50dc84d4fb0bc141f691a5e8184" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35717 } } peers { permanent_uuid: "11db10e937ef426c9344e815d3e7262c" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44619 } } peers { permanent_uuid: "403faea0cec54c4ba72e8997f14141f2" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36381 } }
I20240507 02:07:55.902071 7921 raft_consensus.cc:374] T 7aa9dc252ae84644ae1084ee397d92de P 11db10e937ef426c9344e815d3e7262c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:07:55.902087 7921 raft_consensus.cc:729] T 7aa9dc252ae84644ae1084ee397d92de P 11db10e937ef426c9344e815d3e7262c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11db10e937ef426c9344e815d3e7262c, State: Initialized, Role: FOLLOWER
I20240507 02:07:55.902151 7921 consensus_queue.cc:260] T 7aa9dc252ae84644ae1084ee397d92de P 11db10e937ef426c9344e815d3e7262c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b442a50dc84d4fb0bc141f691a5e8184" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35717 } } peers { permanent_uuid: "11db10e937ef426c9344e815d3e7262c" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44619 } } peers { permanent_uuid: "403faea0cec54c4ba72e8997f14141f2" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36381 } }
I20240507 02:07:55.902334 7921 ts_tablet_manager.cc:1398] T 7aa9dc252ae84644ae1084ee397d92de P 11db10e937ef426c9344e815d3e7262c: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240507 02:07:55.902400 7750 heartbeater.cc:503] Master 127.31.131.126:43871 was elected leader, sending a full tablet report...
W20240507 02:07:55.943653 7751 tablet.cc:2369] T 7aa9dc252ae84644ae1084ee397d92de P 11db10e937ef426c9344e815d3e7262c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240507 02:07:55.962219 7927 raft_consensus.cc:482] T 7aa9dc252ae84644ae1084ee397d92de P 11db10e937ef426c9344e815d3e7262c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 02:07:55.962333 7927 raft_consensus.cc:504] T 7aa9dc252ae84644ae1084ee397d92de P 11db10e937ef426c9344e815d3e7262c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b442a50dc84d4fb0bc141f691a5e8184" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35717 } } peers { permanent_uuid: "11db10e937ef426c9344e815d3e7262c" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44619 } } peers { permanent_uuid: "403faea0cec54c4ba72e8997f14141f2" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36381 } }
I20240507 02:07:55.962685 7927 leader_election.cc:288] T 7aa9dc252ae84644ae1084ee397d92de P 11db10e937ef426c9344e815d3e7262c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b442a50dc84d4fb0bc141f691a5e8184 (127.31.131.65:35717), 403faea0cec54c4ba72e8997f14141f2 (127.31.131.67:36381)
I20240507 02:07:55.964653 7641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:55.962844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.65:35717 (local address 127.31.131.66:58193)
0507 02:07:55.962897 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:55.962909 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:55.962950 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:55.963114 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:55.963115 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:55.963117 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:07:55.963292 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:55.963294 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:55.963916 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:55.963917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:55.964455 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:55.964456 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:55.964518 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:55.964522 (+ 4us) client_negotiation.cc:769] Sending connection context
0507 02:07:55.964549 (+ 27us) client_negotiation.cc:240] Negotiation successful
0507 02:07:55.964582 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20240507 02:07:55.964850 7928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:55.962945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.67:36381 (local address 127.31.131.66:52847)
0507 02:07:55.963112 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:55.963120 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:55.963176 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:55.963286 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:55.963286 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:55.963288 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:07:55.963479 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:55.963481 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:55.964094 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:55.964095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:55.964647 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:55.964649 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:55.964710 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:55.964714 (+ 4us) client_negotiation.cc:769] Sending connection context
0507 02:07:55.964741 (+ 27us) client_negotiation.cc:240] Negotiation successful
0507 02:07:55.964771 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":103,"threads_started":1}
I20240507 02:07:55.965075 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:55.962935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.66:58193 (local address 127.31.131.65:35717)
0507 02:07:55.962982 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:55.962985 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:55.962994 (+ 9us) server_negotiation.cc:407] Connection header received
0507 02:07:55.963022 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:55.963023 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:55.963033 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:55.963098 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:07:55.963369 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:55.963843 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:55.964534 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:55.964977 (+ 443us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:55.964985 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:55.965007 (+ 22us) server_negotiation.cc:299] Negotiation successful
0507 02:07:55.965030 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20240507 02:07:55.965262 7918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:55.963053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.66:52847 (local address 127.31.131.67:36381)
0507 02:07:55.963112 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:55.963114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:55.963181 (+ 67us) server_negotiation.cc:407] Connection header received
0507 02:07:55.963228 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:55.963229 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:55.963237 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:55.963279 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:07:55.963545 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:55.964011 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:55.964727 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:55.965142 (+ 415us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:55.965150 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:55.965173 (+ 23us) server_negotiation.cc:299] Negotiation successful
0507 02:07:55.965202 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20240507 02:07:55.965406 7543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aa9dc252ae84644ae1084ee397d92de" candidate_uuid: "11db10e937ef426c9344e815d3e7262c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b442a50dc84d4fb0bc141f691a5e8184" is_pre_election: true
I20240507 02:07:55.965560 7543 raft_consensus.cc:2454] T 7aa9dc252ae84644ae1084ee397d92de P b442a50dc84d4fb0bc141f691a5e8184 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11db10e937ef426c9344e815d3e7262c in term 0.
I20240507 02:07:55.965543 7844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aa9dc252ae84644ae1084ee397d92de" candidate_uuid: "11db10e937ef426c9344e815d3e7262c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "403faea0cec54c4ba72e8997f14141f2" is_pre_election: true
I20240507 02:07:55.965660 7844 raft_consensus.cc:2454] T 7aa9dc252ae84644ae1084ee397d92de P 403faea0cec54c4ba72e8997f14141f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11db10e937ef426c9344e815d3e7262c in term 0.
I20240507 02:07:55.965765 7636 leader_election.cc:302] T 7aa9dc252ae84644ae1084ee397d92de P 11db10e937ef426c9344e815d3e7262c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11db10e937ef426c9344e815d3e7262c, b442a50dc84d4fb0bc141f691a5e8184; no voters:
I20240507 02:07:55.965905 7927 raft_consensus.cc:2781] T 7aa9dc252ae84644ae1084ee397d92de P 11db10e937ef426c9344e815d3e7262c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 02:07:55.965960 7927 raft_consensus.cc:482] T 7aa9dc252ae84644ae1084ee397d92de P 11db10e937ef426c9344e815d3e7262c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 02:07:55.965981 7927 raft_consensus.cc:3037] T 7aa9dc252ae84644ae1084ee397d92de P 11db10e937ef426c9344e815d3e7262c [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:07:55.966632 7927 raft_consensus.cc:504] T 7aa9dc252ae84644ae1084ee397d92de P 11db10e937ef426c9344e815d3e7262c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b442a50dc84d4fb0bc141f691a5e8184" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35717 } } peers { permanent_uuid: "11db10e937ef426c9344e815d3e7262c" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44619 } } peers { permanent_uuid: "403faea0cec54c4ba72e8997f14141f2" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36381 } }
I20240507 02:07:55.966756 7927 leader_election.cc:288] T 7aa9dc252ae84644ae1084ee397d92de P 11db10e937ef426c9344e815d3e7262c [CANDIDATE]: Term 1 election: Requested vote from peers b442a50dc84d4fb0bc141f691a5e8184 (127.31.131.65:35717), 403faea0cec54c4ba72e8997f14141f2 (127.31.131.67:36381)
I20240507 02:07:55.966907 7543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aa9dc252ae84644ae1084ee397d92de" candidate_uuid: "11db10e937ef426c9344e815d3e7262c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b442a50dc84d4fb0bc141f691a5e8184"
I20240507 02:07:55.966928 7844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aa9dc252ae84644ae1084ee397d92de" candidate_uuid: "11db10e937ef426c9344e815d3e7262c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "403faea0cec54c4ba72e8997f14141f2"
I20240507 02:07:55.966971 7543 raft_consensus.cc:3037] T 7aa9dc252ae84644ae1084ee397d92de P b442a50dc84d4fb0bc141f691a5e8184 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:07:55.966989 7844 raft_consensus.cc:3037] T 7aa9dc252ae84644ae1084ee397d92de P 403faea0cec54c4ba72e8997f14141f2 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:07:55.967592 7543 raft_consensus.cc:2454] T 7aa9dc252ae84644ae1084ee397d92de P b442a50dc84d4fb0bc141f691a5e8184 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11db10e937ef426c9344e815d3e7262c in term 1.
I20240507 02:07:55.967624 7844 raft_consensus.cc:2454] T 7aa9dc252ae84644ae1084ee397d92de P 403faea0cec54c4ba72e8997f14141f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11db10e937ef426c9344e815d3e7262c in term 1.
I20240507 02:07:55.967762 7636 leader_election.cc:302] T 7aa9dc252ae84644ae1084ee397d92de P 11db10e937ef426c9344e815d3e7262c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11db10e937ef426c9344e815d3e7262c, 403faea0cec54c4ba72e8997f14141f2; no voters:
I20240507 02:07:55.967844 7927 raft_consensus.cc:2781] T 7aa9dc252ae84644ae1084ee397d92de P 11db10e937ef426c9344e815d3e7262c [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:07:55.967986 7927 raft_consensus.cc:686] T 7aa9dc252ae84644ae1084ee397d92de P 11db10e937ef426c9344e815d3e7262c [term 1 LEADER]: Becoming Leader. State: Replica: 11db10e937ef426c9344e815d3e7262c, State: Running, Role: LEADER
I20240507 02:07:55.968088 7927 consensus_queue.cc:237] T 7aa9dc252ae84644ae1084ee397d92de P 11db10e937ef426c9344e815d3e7262c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b442a50dc84d4fb0bc141f691a5e8184" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35717 } } peers { permanent_uuid: "11db10e937ef426c9344e815d3e7262c" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44619 } } peers { permanent_uuid: "403faea0cec54c4ba72e8997f14141f2" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36381 } }
I20240507 02:07:55.968798 7403 catalog_manager.cc:5484] T 7aa9dc252ae84644ae1084ee397d92de P 11db10e937ef426c9344e815d3e7262c reported cstate change: term changed from 0 to 1, leader changed from <none> to 11db10e937ef426c9344e815d3e7262c (127.31.131.66). New cstate: current_term: 1 leader_uuid: "11db10e937ef426c9344e815d3e7262c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b442a50dc84d4fb0bc141f691a5e8184" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11db10e937ef426c9344e815d3e7262c" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "403faea0cec54c4ba72e8997f14141f2" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36381 } health_report { overall_health: UNKNOWN } } }
I20240507 02:07:55.981777 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:55.979969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56862 (local address 127.31.131.66:44619)
0507 02:07:55.980020 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:55.980022 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:55.980029 (+ 7us) server_negotiation.cc:407] Connection header received
0507 02:07:55.980063 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:55.980063 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:55.980078 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:55.980138 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0507 02:07:55.980375 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:55.980852 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:55.981354 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:55.981464 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:55.981498 (+ 34us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0507 02:07:55.981579 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0507 02:07:55.981632 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:55.981681 (+ 49us) server_negotiation.cc:299] Negotiation successful
0507 02:07:55.981700 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20240507 02:07:55.983745 7543 raft_consensus.cc:1261] T 7aa9dc252ae84644ae1084ee397d92de P b442a50dc84d4fb0bc141f691a5e8184 [term 1 FOLLOWER]: Refusing update from remote peer 11db10e937ef426c9344e815d3e7262c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 02:07:55.983743 7844 raft_consensus.cc:1261] T 7aa9dc252ae84644ae1084ee397d92de P 403faea0cec54c4ba72e8997f14141f2 [term 1 FOLLOWER]: Refusing update from remote peer 11db10e937ef426c9344e815d3e7262c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 02:07:55.984055 7927 consensus_queue.cc:1035] T 7aa9dc252ae84644ae1084ee397d92de P 11db10e937ef426c9344e815d3e7262c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b442a50dc84d4fb0bc141f691a5e8184" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 02:07:55.984182 7929 consensus_queue.cc:1035] T 7aa9dc252ae84644ae1084ee397d92de P 11db10e937ef426c9344e815d3e7262c [LEADER]: Connected to new peer: Peer: permanent_uuid: "403faea0cec54c4ba72e8997f14141f2" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 02:07:55.987112 7933 mvcc.cc:205] Tried to move back new op lower bound from 7024835280826961920 to 7024835280765747200. Current Snapshot: MvccSnapshot[applied={T|T < 7024835280826961920 or (T in {7024835280826961920})}]
I20240507 02:07:55.988194 7403 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46408:
table { table_name: "test-table" } modify_external_catalogs: true
I20240507 02:07:55.988276 7403 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46408:
table { table_name: "test-table" } modify_external_catalogs: true
I20240507 02:07:55.988956 7403 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5bf6e9ec74044a588b71bc5772a5d06d: Sending DeleteTablet for 3 replicas of tablet 7aa9dc252ae84644ae1084ee397d92de
I20240507 02:07:55.989264 7824 tablet_service.cc:1514] Processing DeleteTablet for tablet 7aa9dc252ae84644ae1084ee397d92de with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:07:55 UTC) from {username='oryx', principal='oryx/127.31.131.126@KRBTEST.COM'} at 127.0.0.1:57052
I20240507 02:07:55.989281 7531 tablet_service.cc:1514] Processing DeleteTablet for tablet 7aa9dc252ae84644ae1084ee397d92de with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:07:55 UTC) from {username='oryx', principal='oryx/127.31.131.126@KRBTEST.COM'} at 127.0.0.1:51620
I20240507 02:07:55.989342 7683 tablet_service.cc:1514] Processing DeleteTablet for tablet 7aa9dc252ae84644ae1084ee397d92de with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:07:55 UTC) from {username='oryx', principal='oryx/127.31.131.126@KRBTEST.COM'} at 127.0.0.1:56860
I20240507 02:07:55.989583 7942 tablet_replica.cc:330] T 7aa9dc252ae84644ae1084ee397d92de P b442a50dc84d4fb0bc141f691a5e8184: stopping tablet replica
I20240507 02:07:55.989713 7941 tablet_replica.cc:330] T 7aa9dc252ae84644ae1084ee397d92de P 403faea0cec54c4ba72e8997f14141f2: stopping tablet replica
I20240507 02:07:55.989682 7403 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46408:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 02:07:55.989790 7941 raft_consensus.cc:2229] T 7aa9dc252ae84644ae1084ee397d92de P 403faea0cec54c4ba72e8997f14141f2 [term 1 FOLLOWER]: Raft consensus shutting down.
W20240507 02:07:55.989816 7403 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 02:07:55.989873 7941 raft_consensus.cc:2258] T 7aa9dc252ae84644ae1084ee397d92de P 403faea0cec54c4ba72e8997f14141f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 02:07:55.989984 7942 raft_consensus.cc:2229] T 7aa9dc252ae84644ae1084ee397d92de P b442a50dc84d4fb0bc141f691a5e8184 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 02:07:55.990067 7942 raft_consensus.cc:2258] T 7aa9dc252ae84644ae1084ee397d92de P b442a50dc84d4fb0bc141f691a5e8184 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 02:07:55.990382 7941 ts_tablet_manager.cc:1875] T 7aa9dc252ae84644ae1084ee397d92de P 403faea0cec54c4ba72e8997f14141f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 02:07:55.990463 7942 ts_tablet_manager.cc:1875] T 7aa9dc252ae84644ae1084ee397d92de P b442a50dc84d4fb0bc141f691a5e8184: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 02:07:55.991132 7683 tablet_service.cc:1467] Processing CreateTablet for tablet 1a8af0a6630f48348c55b83cf96426e9 (DEFAULT_TABLE table=test-table [id=f5e9a56872b1400a9db999b210c77620]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:07:55.991221 7531 tablet_service.cc:1467] Processing CreateTablet for tablet 1a8af0a6630f48348c55b83cf96426e9 (DEFAULT_TABLE table=test-table [id=f5e9a56872b1400a9db999b210c77620]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:07:55.991252 7683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a8af0a6630f48348c55b83cf96426e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:07:55.991405 7531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a8af0a6630f48348c55b83cf96426e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:07:55.991523 7943 tablet_replica.cc:330] T 7aa9dc252ae84644ae1084ee397d92de P 11db10e937ef426c9344e815d3e7262c: stopping tablet replica
I20240507 02:07:55.991587 7943 raft_consensus.cc:2229] T 7aa9dc252ae84644ae1084ee397d92de P 11db10e937ef426c9344e815d3e7262c [term 1 LEADER]: Raft consensus shutting down.
I20240507 02:07:55.991674 7943 raft_consensus.cc:2258] T 7aa9dc252ae84644ae1084ee397d92de P 11db10e937ef426c9344e815d3e7262c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 02:07:55.992283 7942 ts_tablet_manager.cc:1888] T 7aa9dc252ae84644ae1084ee397d92de P b442a50dc84d4fb0bc141f691a5e8184: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240507 02:07:55.992334 7942 log.cc:1196] T 7aa9dc252ae84644ae1084ee397d92de P b442a50dc84d4fb0bc141f691a5e8184: Deleting WAL directory at /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/ts-0/wal/wals/7aa9dc252ae84644ae1084ee397d92de
I20240507 02:07:55.992568 7942 ts_tablet_manager.cc:1909] T 7aa9dc252ae84644ae1084ee397d92de P b442a50dc84d4fb0bc141f691a5e8184: Deleting consensus metadata
I20240507 02:07:55.992579 7921 tablet_bootstrap.cc:492] T 1a8af0a6630f48348c55b83cf96426e9 P 11db10e937ef426c9344e815d3e7262c: Bootstrap starting.
I20240507 02:07:55.992580 7919 tablet_bootstrap.cc:492] T 1a8af0a6630f48348c55b83cf96426e9 P b442a50dc84d4fb0bc141f691a5e8184: Bootstrap starting.
I20240507 02:07:55.992928 7921 tablet_bootstrap.cc:654] T 1a8af0a6630f48348c55b83cf96426e9 P 11db10e937ef426c9344e815d3e7262c: Neither blocks nor log segments found. Creating new log.
I20240507 02:07:55.993048 7919 tablet_bootstrap.cc:654] T 1a8af0a6630f48348c55b83cf96426e9 P b442a50dc84d4fb0bc141f691a5e8184: Neither blocks nor log segments found. Creating new log.
I20240507 02:07:55.993312 7390 catalog_manager.cc:4830] TS b442a50dc84d4fb0bc141f691a5e8184 (127.31.131.65:35717): tablet 7aa9dc252ae84644ae1084ee397d92de (table test-table [id=c66f119023db4092b1547ed08c66aefa]) successfully deleted
I20240507 02:07:55.993433 7824 tablet_service.cc:1467] Processing CreateTablet for tablet 1a8af0a6630f48348c55b83cf96426e9 (DEFAULT_TABLE table=test-table [id=f5e9a56872b1400a9db999b210c77620]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:07:55.993495 7921 tablet_bootstrap.cc:492] T 1a8af0a6630f48348c55b83cf96426e9 P 11db10e937ef426c9344e815d3e7262c: No bootstrap required, opened a new log
I20240507 02:07:55.993526 7824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a8af0a6630f48348c55b83cf96426e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:07:55.993528 7921 ts_tablet_manager.cc:1367] T 1a8af0a6630f48348c55b83cf96426e9 P 11db10e937ef426c9344e815d3e7262c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240507 02:07:55.993676 7921 raft_consensus.cc:348] T 1a8af0a6630f48348c55b83cf96426e9 P 11db10e937ef426c9344e815d3e7262c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b442a50dc84d4fb0bc141f691a5e8184" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35717 } } peers { permanent_uuid: "11db10e937ef426c9344e815d3e7262c" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44619 } } peers { permanent_uuid: "403faea0cec54c4ba72e8997f14141f2" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36381 } }
I20240507 02:07:55.993731 7921 raft_consensus.cc:374] T 1a8af0a6630f48348c55b83cf96426e9 P 11db10e937ef426c9344e815d3e7262c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:07:55.993748 7921 raft_consensus.cc:729] T 1a8af0a6630f48348c55b83cf96426e9 P 11db10e937ef426c9344e815d3e7262c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11db10e937ef426c9344e815d3e7262c, State: Initialized, Role: FOLLOWER
I20240507 02:07:55.993783 7921 consensus_queue.cc:260] T 1a8af0a6630f48348c55b83cf96426e9 P 11db10e937ef426c9344e815d3e7262c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b442a50dc84d4fb0bc141f691a5e8184" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35717 } } peers { permanent_uuid: "11db10e937ef426c9344e815d3e7262c" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44619 } } peers { permanent_uuid: "403faea0cec54c4ba72e8997f14141f2" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36381 } }
I20240507 02:07:55.993852 7921 ts_tablet_manager.cc:1398] T 1a8af0a6630f48348c55b83cf96426e9 P 11db10e937ef426c9344e815d3e7262c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:55.994625 7943 ts_tablet_manager.cc:1875] T 7aa9dc252ae84644ae1084ee397d92de P 11db10e937ef426c9344e815d3e7262c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 02:07:55.994875 7920 tablet_bootstrap.cc:492] T 1a8af0a6630f48348c55b83cf96426e9 P 403faea0cec54c4ba72e8997f14141f2: Bootstrap starting.
I20240507 02:07:55.995373 7941 ts_tablet_manager.cc:1888] T 7aa9dc252ae84644ae1084ee397d92de P 403faea0cec54c4ba72e8997f14141f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240507 02:07:55.995400 7920 tablet_bootstrap.cc:654] T 1a8af0a6630f48348c55b83cf96426e9 P 403faea0cec54c4ba72e8997f14141f2: Neither blocks nor log segments found. Creating new log.
I20240507 02:07:55.995429 7941 log.cc:1196] T 7aa9dc252ae84644ae1084ee397d92de P 403faea0cec54c4ba72e8997f14141f2: Deleting WAL directory at /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/ts-2/wal/wals/7aa9dc252ae84644ae1084ee397d92de
I20240507 02:07:55.995661 7941 ts_tablet_manager.cc:1909] T 7aa9dc252ae84644ae1084ee397d92de P 403faea0cec54c4ba72e8997f14141f2: Deleting consensus metadata
I20240507 02:07:55.995960 7390 catalog_manager.cc:4830] TS 403faea0cec54c4ba72e8997f14141f2 (127.31.131.67:36381): tablet 7aa9dc252ae84644ae1084ee397d92de (table test-table [id=c66f119023db4092b1547ed08c66aefa]) successfully deleted
I20240507 02:07:55.996037 7943 ts_tablet_manager.cc:1888] T 7aa9dc252ae84644ae1084ee397d92de P 11db10e937ef426c9344e815d3e7262c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240507 02:07:55.996131 7943 log.cc:1196] T 7aa9dc252ae84644ae1084ee397d92de P 11db10e937ef426c9344e815d3e7262c: Deleting WAL directory at /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/ts-1/wal/wals/7aa9dc252ae84644ae1084ee397d92de
I20240507 02:07:55.996387 7943 ts_tablet_manager.cc:1909] T 7aa9dc252ae84644ae1084ee397d92de P 11db10e937ef426c9344e815d3e7262c: Deleting consensus metadata
I20240507 02:07:55.996397 7920 tablet_bootstrap.cc:492] T 1a8af0a6630f48348c55b83cf96426e9 P 403faea0cec54c4ba72e8997f14141f2: No bootstrap required, opened a new log
I20240507 02:07:55.996435 7920 ts_tablet_manager.cc:1367] T 1a8af0a6630f48348c55b83cf96426e9 P 403faea0cec54c4ba72e8997f14141f2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240507 02:07:55.996591 7920 raft_consensus.cc:348] T 1a8af0a6630f48348c55b83cf96426e9 P 403faea0cec54c4ba72e8997f14141f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b442a50dc84d4fb0bc141f691a5e8184" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35717 } } peers { permanent_uuid: "11db10e937ef426c9344e815d3e7262c" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44619 } } peers { permanent_uuid: "403faea0cec54c4ba72e8997f14141f2" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36381 } }
I20240507 02:07:55.996646 7920 raft_consensus.cc:374] T 1a8af0a6630f48348c55b83cf96426e9 P 403faea0cec54c4ba72e8997f14141f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:07:55.996665 7920 raft_consensus.cc:729] T 1a8af0a6630f48348c55b83cf96426e9 P 403faea0cec54c4ba72e8997f14141f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 403faea0cec54c4ba72e8997f14141f2, State: Initialized, Role: FOLLOWER
I20240507 02:07:55.996688 7389 catalog_manager.cc:4830] TS 11db10e937ef426c9344e815d3e7262c (127.31.131.66:44619): tablet 7aa9dc252ae84644ae1084ee397d92de (table test-table [id=c66f119023db4092b1547ed08c66aefa]) successfully deleted
I20240507 02:07:55.996706 7920 consensus_queue.cc:260] T 1a8af0a6630f48348c55b83cf96426e9 P 403faea0cec54c4ba72e8997f14141f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b442a50dc84d4fb0bc141f691a5e8184" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35717 } } peers { permanent_uuid: "11db10e937ef426c9344e815d3e7262c" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44619 } } peers { permanent_uuid: "403faea0cec54c4ba72e8997f14141f2" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36381 } }
I20240507 02:07:55.996783 7920 ts_tablet_manager.cc:1398] T 1a8af0a6630f48348c55b83cf96426e9 P 403faea0cec54c4ba72e8997f14141f2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:56.002490 7919 tablet_bootstrap.cc:492] T 1a8af0a6630f48348c55b83cf96426e9 P b442a50dc84d4fb0bc141f691a5e8184: No bootstrap required, opened a new log
I20240507 02:07:56.002565 7919 ts_tablet_manager.cc:1367] T 1a8af0a6630f48348c55b83cf96426e9 P b442a50dc84d4fb0bc141f691a5e8184: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20240507 02:07:56.002707 7919 raft_consensus.cc:348] T 1a8af0a6630f48348c55b83cf96426e9 P b442a50dc84d4fb0bc141f691a5e8184 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b442a50dc84d4fb0bc141f691a5e8184" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35717 } } peers { permanent_uuid: "11db10e937ef426c9344e815d3e7262c" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44619 } } peers { permanent_uuid: "403faea0cec54c4ba72e8997f14141f2" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36381 } }
I20240507 02:07:56.002761 7919 raft_consensus.cc:374] T 1a8af0a6630f48348c55b83cf96426e9 P b442a50dc84d4fb0bc141f691a5e8184 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:07:56.002774 7919 raft_consensus.cc:729] T 1a8af0a6630f48348c55b83cf96426e9 P b442a50dc84d4fb0bc141f691a5e8184 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b442a50dc84d4fb0bc141f691a5e8184, State: Initialized, Role: FOLLOWER
I20240507 02:07:56.002805 7919 consensus_queue.cc:260] T 1a8af0a6630f48348c55b83cf96426e9 P b442a50dc84d4fb0bc141f691a5e8184 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b442a50dc84d4fb0bc141f691a5e8184" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35717 } } peers { permanent_uuid: "11db10e937ef426c9344e815d3e7262c" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44619 } } peers { permanent_uuid: "403faea0cec54c4ba72e8997f14141f2" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36381 } }
I20240507 02:07:56.002888 7919 ts_tablet_manager.cc:1398] T 1a8af0a6630f48348c55b83cf96426e9 P b442a50dc84d4fb0bc141f691a5e8184: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20240507 02:07:56.051697 7608 tablet.cc:2369] T 1a8af0a6630f48348c55b83cf96426e9 P b442a50dc84d4fb0bc141f691a5e8184: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240507 02:07:56.099999 7892 tablet.cc:2369] T 1a8af0a6630f48348c55b83cf96426e9 P 403faea0cec54c4ba72e8997f14141f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240507 02:07:56.217619 7925 raft_consensus.cc:482] T 1a8af0a6630f48348c55b83cf96426e9 P b442a50dc84d4fb0bc141f691a5e8184 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 02:07:56.217722 7925 raft_consensus.cc:504] T 1a8af0a6630f48348c55b83cf96426e9 P b442a50dc84d4fb0bc141f691a5e8184 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b442a50dc84d4fb0bc141f691a5e8184" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35717 } } peers { permanent_uuid: "11db10e937ef426c9344e815d3e7262c" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44619 } } peers { permanent_uuid: "403faea0cec54c4ba72e8997f14141f2" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36381 } }
I20240507 02:07:56.218061 7925 leader_election.cc:288] T 1a8af0a6630f48348c55b83cf96426e9 P b442a50dc84d4fb0bc141f691a5e8184 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 11db10e937ef426c9344e815d3e7262c (127.31.131.66:44619), 403faea0cec54c4ba72e8997f14141f2 (127.31.131.67:36381)
I20240507 02:07:56.219645 7924 raft_consensus.cc:482] T 1a8af0a6630f48348c55b83cf96426e9 P 403faea0cec54c4ba72e8997f14141f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 02:07:56.219717 7924 raft_consensus.cc:504] T 1a8af0a6630f48348c55b83cf96426e9 P 403faea0cec54c4ba72e8997f14141f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b442a50dc84d4fb0bc141f691a5e8184" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35717 } } peers { permanent_uuid: "11db10e937ef426c9344e815d3e7262c" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44619 } } peers { permanent_uuid: "403faea0cec54c4ba72e8997f14141f2" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36381 } }
I20240507 02:07:56.219971 7924 leader_election.cc:288] T 1a8af0a6630f48348c55b83cf96426e9 P 403faea0cec54c4ba72e8997f14141f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b442a50dc84d4fb0bc141f691a5e8184 (127.31.131.65:35717), 11db10e937ef426c9344e815d3e7262c (127.31.131.66:44619)
I20240507 02:07:56.221371 7918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:56.218437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:36831 (local address 127.31.131.67:36381)
0507 02:07:56.218474 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:56.218476 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:56.218485 (+ 9us) server_negotiation.cc:407] Connection header received
0507 02:07:56.218514 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:56.218515 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:56.218525 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:56.218607 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:07:56.218902 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:56.219635 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:56.220709 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:56.221150 (+ 441us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:56.221158 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:56.221273 (+ 115us) server_negotiation.cc:299] Negotiation successful
0507 02:07:56.221309 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20240507 02:07:56.221397 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:56.218383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:40571 (local address 127.31.131.66:44619)
0507 02:07:56.218445 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:56.218447 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:56.218493 (+ 46us) server_negotiation.cc:407] Connection header received
0507 02:07:56.218592 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:56.218593 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:56.218605 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:56.218642 (+ 37us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:07:56.218992 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:56.219664 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:56.220787 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:56.221203 (+ 416us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:56.221209 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:56.221312 (+ 103us) server_negotiation.cc:299] Negotiation successful
0507 02:07:56.221345 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20240507 02:07:56.221596 7948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:56.218268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.66:44619 (local address 127.31.131.65:40571)
0507 02:07:56.218433 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:56.218444 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:56.218511 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:56.218680 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:56.218681 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:56.218683 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:07:56.218911 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:56.218913 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:56.219739 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:56.219740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:56.220709 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:56.220710 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:56.221231 (+ 521us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:56.221233 (+ 2us) client_negotiation.cc:769] Sending connection context
0507 02:07:56.221484 (+ 251us) client_negotiation.cc:240] Negotiation successful
0507 02:07:56.221526 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":78,"threads_started":1}
I20240507 02:07:56.221916 7947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:56.218250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.67:36381 (local address 127.31.131.65:36831)
0507 02:07:56.218382 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:56.218392 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:56.218452 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:56.218610 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:56.218611 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:56.218613 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:07:56.218835 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:56.218838 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:56.219726 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:56.219727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:56.220623 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:56.220625 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:56.221186 (+ 561us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:56.221190 (+ 4us) client_negotiation.cc:769] Sending connection context
0507 02:07:56.221806 (+ 616us) client_negotiation.cc:240] Negotiation successful
0507 02:07:56.221847 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":65,"threads_started":1}
I20240507 02:07:56.221855 7703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a8af0a6630f48348c55b83cf96426e9" candidate_uuid: "b442a50dc84d4fb0bc141f691a5e8184" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11db10e937ef426c9344e815d3e7262c" is_pre_election: true
I20240507 02:07:56.221978 7703 raft_consensus.cc:2454] T 1a8af0a6630f48348c55b83cf96426e9 P 11db10e937ef426c9344e815d3e7262c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b442a50dc84d4fb0bc141f691a5e8184 in term 0.
I20240507 02:07:56.222157 7494 leader_election.cc:302] T 1a8af0a6630f48348c55b83cf96426e9 P b442a50dc84d4fb0bc141f691a5e8184 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11db10e937ef426c9344e815d3e7262c, b442a50dc84d4fb0bc141f691a5e8184; no voters:
I20240507 02:07:56.222275 7925 raft_consensus.cc:2781] T 1a8af0a6630f48348c55b83cf96426e9 P b442a50dc84d4fb0bc141f691a5e8184 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 02:07:56.222321 7925 raft_consensus.cc:482] T 1a8af0a6630f48348c55b83cf96426e9 P b442a50dc84d4fb0bc141f691a5e8184 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 02:07:56.222338 7925 raft_consensus.cc:3037] T 1a8af0a6630f48348c55b83cf96426e9 P b442a50dc84d4fb0bc141f691a5e8184 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:07:56.222292 7844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a8af0a6630f48348c55b83cf96426e9" candidate_uuid: "b442a50dc84d4fb0bc141f691a5e8184" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "403faea0cec54c4ba72e8997f14141f2" is_pre_election: true
I20240507 02:07:56.222363 7844 raft_consensus.cc:2454] T 1a8af0a6630f48348c55b83cf96426e9 P 403faea0cec54c4ba72e8997f14141f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b442a50dc84d4fb0bc141f691a5e8184 in term 0.
I20240507 02:07:56.222433 7783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:56.220112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.65:35717 (local address 127.31.131.67:50479)
0507 02:07:56.220153 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:56.220164 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:56.220201 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:56.220384 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:56.220385 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:56.220387 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:07:56.220559 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:56.220561 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:56.221373 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:56.221374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:56.222253 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:56.222255 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:56.222306 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:56.222309 (+ 3us) client_negotiation.cc:769] Sending connection context
0507 02:07:56.222335 (+ 26us) client_negotiation.cc:240] Negotiation successful
0507 02:07:56.222369 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20240507 02:07:56.223163 7949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:56.220419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.66:44619 (local address 127.31.131.67:35099)
0507 02:07:56.220679 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:56.220688 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:56.220764 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:56.220920 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:56.220921 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:56.220922 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:07:56.221169 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:56.221170 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:56.222041 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:56.222042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:56.222908 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:56.222911 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:56.222994 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:56.222998 (+ 4us) client_negotiation.cc:769] Sending connection context
0507 02:07:56.223064 (+ 66us) client_negotiation.cc:240] Negotiation successful
0507 02:07:56.223096 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":48,"threads_started":1}
I20240507 02:07:56.223147 7925 raft_consensus.cc:504] T 1a8af0a6630f48348c55b83cf96426e9 P b442a50dc84d4fb0bc141f691a5e8184 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b442a50dc84d4fb0bc141f691a5e8184" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35717 } } peers { permanent_uuid: "11db10e937ef426c9344e815d3e7262c" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44619 } } peers { permanent_uuid: "403faea0cec54c4ba72e8997f14141f2" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36381 } }
I20240507 02:07:56.223148 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:56.220219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.67:50479 (local address 127.31.131.65:35717)
0507 02:07:56.220263 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:56.220264 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:56.220271 (+ 7us) server_negotiation.cc:407] Connection header received
0507 02:07:56.220288 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:56.220289 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:56.220296 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:56.220342 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:07:56.220640 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:56.221300 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:56.222637 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:56.223038 (+ 401us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:56.223045 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:56.223064 (+ 19us) server_negotiation.cc:299] Negotiation successful
0507 02:07:56.223091 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20240507 02:07:56.223322 7925 leader_election.cc:288] T 1a8af0a6630f48348c55b83cf96426e9 P b442a50dc84d4fb0bc141f691a5e8184 [CANDIDATE]: Term 1 election: Requested vote from peers 11db10e937ef426c9344e815d3e7262c (127.31.131.66:44619), 403faea0cec54c4ba72e8997f14141f2 (127.31.131.67:36381)
I20240507 02:07:56.223383 7543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a8af0a6630f48348c55b83cf96426e9" candidate_uuid: "403faea0cec54c4ba72e8997f14141f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b442a50dc84d4fb0bc141f691a5e8184" is_pre_election: true
I20240507 02:07:56.223460 7543 raft_consensus.cc:2379] T 1a8af0a6630f48348c55b83cf96426e9 P b442a50dc84d4fb0bc141f691a5e8184 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 403faea0cec54c4ba72e8997f14141f2 in current term 1: Already voted for candidate b442a50dc84d4fb0bc141f691a5e8184 in this term.
I20240507 02:07:56.223554 7844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a8af0a6630f48348c55b83cf96426e9" candidate_uuid: "b442a50dc84d4fb0bc141f691a5e8184" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "403faea0cec54c4ba72e8997f14141f2"
I20240507 02:07:56.223585 7950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:56.220539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.67:35099 (local address 127.31.131.66:44619)
0507 02:07:56.220633 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:56.220635 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:56.220743 (+ 108us) server_negotiation.cc:407] Connection header received
0507 02:07:56.220828 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:56.220829 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:56.220837 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:56.220867 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:07:56.221236 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:56.221920 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:56.222995 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:56.223468 (+ 473us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:56.223475 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:56.223498 (+ 23us) server_negotiation.cc:299] Negotiation successful
0507 02:07:56.223528 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79,"thread_start_us":51,"threads_started":1}
I20240507 02:07:56.223615 7844 raft_consensus.cc:3037] T 1a8af0a6630f48348c55b83cf96426e9 P 403faea0cec54c4ba72e8997f14141f2 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:07:56.223745 7703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a8af0a6630f48348c55b83cf96426e9" candidate_uuid: "403faea0cec54c4ba72e8997f14141f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11db10e937ef426c9344e815d3e7262c" is_pre_election: true
I20240507 02:07:56.223779 7702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a8af0a6630f48348c55b83cf96426e9" candidate_uuid: "b442a50dc84d4fb0bc141f691a5e8184" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11db10e937ef426c9344e815d3e7262c"
I20240507 02:07:56.223809 7703 raft_consensus.cc:2454] T 1a8af0a6630f48348c55b83cf96426e9 P 11db10e937ef426c9344e815d3e7262c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 403faea0cec54c4ba72e8997f14141f2 in term 0.
I20240507 02:07:56.223970 7777 leader_election.cc:302] T 1a8af0a6630f48348c55b83cf96426e9 P 403faea0cec54c4ba72e8997f14141f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 11db10e937ef426c9344e815d3e7262c, 403faea0cec54c4ba72e8997f14141f2; no voters: b442a50dc84d4fb0bc141f691a5e8184
I20240507 02:07:56.224275 7844 raft_consensus.cc:2454] T 1a8af0a6630f48348c55b83cf96426e9 P 403faea0cec54c4ba72e8997f14141f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b442a50dc84d4fb0bc141f691a5e8184 in term 1.
I20240507 02:07:56.224339 7924 raft_consensus.cc:2741] T 1a8af0a6630f48348c55b83cf96426e9 P 403faea0cec54c4ba72e8997f14141f2 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20240507 02:07:56.224436 7495 leader_election.cc:302] T 1a8af0a6630f48348c55b83cf96426e9 P b442a50dc84d4fb0bc141f691a5e8184 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 403faea0cec54c4ba72e8997f14141f2, b442a50dc84d4fb0bc141f691a5e8184; no voters: 11db10e937ef426c9344e815d3e7262c
I20240507 02:07:56.224532 7925 raft_consensus.cc:2781] T 1a8af0a6630f48348c55b83cf96426e9 P b442a50dc84d4fb0bc141f691a5e8184 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:07:56.224618 7925 raft_consensus.cc:686] T 1a8af0a6630f48348c55b83cf96426e9 P b442a50dc84d4fb0bc141f691a5e8184 [term 1 LEADER]: Becoming Leader. State: Replica: b442a50dc84d4fb0bc141f691a5e8184, State: Running, Role: LEADER
I20240507 02:07:56.224673 7925 consensus_queue.cc:237] T 1a8af0a6630f48348c55b83cf96426e9 P b442a50dc84d4fb0bc141f691a5e8184 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b442a50dc84d4fb0bc141f691a5e8184" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35717 } } peers { permanent_uuid: "11db10e937ef426c9344e815d3e7262c" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44619 } } peers { permanent_uuid: "403faea0cec54c4ba72e8997f14141f2" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36381 } }
I20240507 02:07:56.225239 7402 catalog_manager.cc:5484] T 1a8af0a6630f48348c55b83cf96426e9 P b442a50dc84d4fb0bc141f691a5e8184 reported cstate change: term changed from 0 to 1, leader changed from <none> to b442a50dc84d4fb0bc141f691a5e8184 (127.31.131.65). New cstate: current_term: 1 leader_uuid: "b442a50dc84d4fb0bc141f691a5e8184" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b442a50dc84d4fb0bc141f691a5e8184" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "11db10e937ef426c9344e815d3e7262c" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "403faea0cec54c4ba72e8997f14141f2" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36381 } health_report { overall_health: UNKNOWN } } }
May 07 02:07:56 dist-test-slave-v0bg krb5kdc[7358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047675, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.131.126@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM
I20240507 02:07:56.300853 7958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:56.295313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:43871 (local address 127.0.0.1:46418)
0507 02:07:56.295444 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:56.295453 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:56.295530 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:56.295686 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:56.295687 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:56.296019 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:07:56.296179 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:56.296181 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:56.296762 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:56.296763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:56.297112 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:56.297113 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:56.297161 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:56.297619 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:07:56.297623 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:07:56.298953 (+ 1330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:07:56.300151 (+ 1198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:56.300155 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:56.300158 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:56.300319 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:56.300434 (+ 115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:56.300434 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:56.300435 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:56.300458 (+ 23us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:56.300604 (+ 146us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:07:56.300607 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:07:56.300641 (+ 34us) client_negotiation.cc:769] Sending connection context
0507 02:07:56.300692 (+ 51us) client_negotiation.cc:240] Negotiation successful
0507 02:07:56.300776 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":73,"threads_started":1}
I20240507 02:07:56.300920 7454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:56.295349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46418 (local address 127.31.131.126:43871)
0507 02:07:56.295408 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:56.295410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:56.295520 (+ 110us) server_negotiation.cc:407] Connection header received
0507 02:07:56.295612 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:56.295613 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:56.295623 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:56.295673 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:56.296237 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:56.296708 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:56.297159 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:56.297264 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:56.299024 (+ 1760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:56.299028 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:56.299028 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:07:56.299034 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:56.300088 (+ 1054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:56.300373 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:56.300374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:56.300374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:56.300389 (+ 15us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:56.300491 (+ 102us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:56.300491 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:56.300492 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:56.300566 (+ 74us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:56.300609 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:56.300711 (+ 102us) server_negotiation.cc:299] Negotiation successful
0507 02:07:56.300825 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":30}
I20240507 02:07:56.302542 7402 catalog_manager.cc:1904] Servicing CreateTable request from {username='oryx', principal='oryx/127.31.131.126@KRBTEST.COM'} at 127.0.0.1:46418:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20240507 02:07:56.302747 7402 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 02:07:56.304019 7683 tablet_service.cc:1467] Processing CreateTablet for tablet 26d6142f80d24395b9bee779d80aff2a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1991d511cdc14e3a83db4ff793ae94c9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20240507 02:07:56.304057 7531 tablet_service.cc:1467] Processing CreateTablet for tablet 26d6142f80d24395b9bee779d80aff2a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1991d511cdc14e3a83db4ff793ae94c9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20240507 02:07:56.304064 7824 tablet_service.cc:1467] Processing CreateTablet for tablet 26d6142f80d24395b9bee779d80aff2a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1991d511cdc14e3a83db4ff793ae94c9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20240507 02:07:56.304168 7824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26d6142f80d24395b9bee779d80aff2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:07:56.304178 7531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26d6142f80d24395b9bee779d80aff2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:07:56.304183 7683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26d6142f80d24395b9bee779d80aff2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:07:56.305166 7919 tablet_bootstrap.cc:492] T 26d6142f80d24395b9bee779d80aff2a P b442a50dc84d4fb0bc141f691a5e8184: Bootstrap starting.
I20240507 02:07:56.305491 7921 tablet_bootstrap.cc:492] T 26d6142f80d24395b9bee779d80aff2a P 11db10e937ef426c9344e815d3e7262c: Bootstrap starting.
I20240507 02:07:56.305498 7919 tablet_bootstrap.cc:654] T 26d6142f80d24395b9bee779d80aff2a P b442a50dc84d4fb0bc141f691a5e8184: Neither blocks nor log segments found. Creating new log.
I20240507 02:07:56.305496 7920 tablet_bootstrap.cc:492] T 26d6142f80d24395b9bee779d80aff2a P 403faea0cec54c4ba72e8997f14141f2: Bootstrap starting.
I20240507 02:07:56.305773 7921 tablet_bootstrap.cc:654] T 26d6142f80d24395b9bee779d80aff2a P 11db10e937ef426c9344e815d3e7262c: Neither blocks nor log segments found. Creating new log.
I20240507 02:07:56.305859 7920 tablet_bootstrap.cc:654] T 26d6142f80d24395b9bee779d80aff2a P 403faea0cec54c4ba72e8997f14141f2: Neither blocks nor log segments found. Creating new log.
I20240507 02:07:56.306315 7921 tablet_bootstrap.cc:492] T 26d6142f80d24395b9bee779d80aff2a P 11db10e937ef426c9344e815d3e7262c: No bootstrap required, opened a new log
I20240507 02:07:56.306365 7921 ts_tablet_manager.cc:1367] T 26d6142f80d24395b9bee779d80aff2a P 11db10e937ef426c9344e815d3e7262c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240507 02:07:56.306484 7920 tablet_bootstrap.cc:492] T 26d6142f80d24395b9bee779d80aff2a P 403faea0cec54c4ba72e8997f14141f2: No bootstrap required, opened a new log
I20240507 02:07:56.306493 7921 raft_consensus.cc:348] T 26d6142f80d24395b9bee779d80aff2a P 11db10e937ef426c9344e815d3e7262c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b442a50dc84d4fb0bc141f691a5e8184" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35717 } } peers { permanent_uuid: "11db10e937ef426c9344e815d3e7262c" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44619 } } peers { permanent_uuid: "403faea0cec54c4ba72e8997f14141f2" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36381 } }
I20240507 02:07:56.306525 7919 tablet_bootstrap.cc:492] T 26d6142f80d24395b9bee779d80aff2a P b442a50dc84d4fb0bc141f691a5e8184: No bootstrap required, opened a new log
I20240507 02:07:56.306533 7920 ts_tablet_manager.cc:1367] T 26d6142f80d24395b9bee779d80aff2a P 403faea0cec54c4ba72e8997f14141f2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240507 02:07:56.306581 7921 raft_consensus.cc:374] T 26d6142f80d24395b9bee779d80aff2a P 11db10e937ef426c9344e815d3e7262c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:07:56.306592 7919 ts_tablet_manager.cc:1367] T 26d6142f80d24395b9bee779d80aff2a P b442a50dc84d4fb0bc141f691a5e8184: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240507 02:07:56.306602 7921 raft_consensus.cc:729] T 26d6142f80d24395b9bee779d80aff2a P 11db10e937ef426c9344e815d3e7262c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11db10e937ef426c9344e815d3e7262c, State: Initialized, Role: FOLLOWER
I20240507 02:07:56.306644 7921 consensus_queue.cc:260] T 26d6142f80d24395b9bee779d80aff2a P 11db10e937ef426c9344e815d3e7262c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b442a50dc84d4fb0bc141f691a5e8184" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35717 } } peers { permanent_uuid: "11db10e937ef426c9344e815d3e7262c" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44619 } } peers { permanent_uuid: "403faea0cec54c4ba72e8997f14141f2" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36381 } }
I20240507 02:07:56.306694 7920 raft_consensus.cc:348] T 26d6142f80d24395b9bee779d80aff2a P 403faea0cec54c4ba72e8997f14141f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b442a50dc84d4fb0bc141f691a5e8184" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35717 } } peers { permanent_uuid: "11db10e937ef426c9344e815d3e7262c" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44619 } } peers { permanent_uuid: "403faea0cec54c4ba72e8997f14141f2" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36381 } }
I20240507 02:07:56.306728 7921 ts_tablet_manager.cc:1398] T 26d6142f80d24395b9bee779d80aff2a P 11db10e937ef426c9344e815d3e7262c: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20240507 02:07:56.306756 7920 raft_consensus.cc:374] T 26d6142f80d24395b9bee779d80aff2a P 403faea0cec54c4ba72e8997f14141f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:07:56.306780 7920 raft_consensus.cc:729] T 26d6142f80d24395b9bee779d80aff2a P 403faea0cec54c4ba72e8997f14141f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 403faea0cec54c4ba72e8997f14141f2, State: Initialized, Role: FOLLOWER
I20240507 02:07:56.306735 7919 raft_consensus.cc:348] T 26d6142f80d24395b9bee779d80aff2a P b442a50dc84d4fb0bc141f691a5e8184 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b442a50dc84d4fb0bc141f691a5e8184" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35717 } } peers { permanent_uuid: "11db10e937ef426c9344e815d3e7262c" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44619 } } peers { permanent_uuid: "403faea0cec54c4ba72e8997f14141f2" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36381 } }
I20240507 02:07:56.306797 7919 raft_consensus.cc:374] T 26d6142f80d24395b9bee779d80aff2a P b442a50dc84d4fb0bc141f691a5e8184 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:07:56.306816 7919 raft_consensus.cc:729] T 26d6142f80d24395b9bee779d80aff2a P b442a50dc84d4fb0bc141f691a5e8184 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b442a50dc84d4fb0bc141f691a5e8184, State: Initialized, Role: FOLLOWER
I20240507 02:07:56.306807 7927 tablet_replica.cc:439] T 26d6142f80d24395b9bee779d80aff2a P 11db10e937ef426c9344e815d3e7262c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b442a50dc84d4fb0bc141f691a5e8184" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35717 } } peers { permanent_uuid: "11db10e937ef426c9344e815d3e7262c" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44619 } } peers { permanent_uuid: "403faea0cec54c4ba72e8997f14141f2" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36381 } } }
I20240507 02:07:56.306821 7920 consensus_queue.cc:260] T 26d6142f80d24395b9bee779d80aff2a P 403faea0cec54c4ba72e8997f14141f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b442a50dc84d4fb0bc141f691a5e8184" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35717 } } peers { permanent_uuid: "11db10e937ef426c9344e815d3e7262c" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44619 } } peers { permanent_uuid: "403faea0cec54c4ba72e8997f14141f2" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36381 } }
I20240507 02:07:56.306856 7927 tablet_replica.cc:442] T 26d6142f80d24395b9bee779d80aff2a P 11db10e937ef426c9344e815d3e7262c: This TxnStatusTablet replica's current role is: FOLLOWER
I20240507 02:07:56.306859 7919 consensus_queue.cc:260] T 26d6142f80d24395b9bee779d80aff2a P b442a50dc84d4fb0bc141f691a5e8184 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b442a50dc84d4fb0bc141f691a5e8184" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35717 } } peers { permanent_uuid: "11db10e937ef426c9344e815d3e7262c" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44619 } } peers { permanent_uuid: "403faea0cec54c4ba72e8997f14141f2" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36381 } }
I20240507 02:07:56.306912 7920 ts_tablet_manager.cc:1398] T 26d6142f80d24395b9bee779d80aff2a P 403faea0cec54c4ba72e8997f14141f2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:56.306946 7919 ts_tablet_manager.cc:1398] T 26d6142f80d24395b9bee779d80aff2a P b442a50dc84d4fb0bc141f691a5e8184: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:56.306941 7924 tablet_replica.cc:439] T 26d6142f80d24395b9bee779d80aff2a P 403faea0cec54c4ba72e8997f14141f2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b442a50dc84d4fb0bc141f691a5e8184" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35717 } } peers { permanent_uuid: "11db10e937ef426c9344e815d3e7262c" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44619 } } peers { permanent_uuid: "403faea0cec54c4ba72e8997f14141f2" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36381 } } }
I20240507 02:07:56.307024 7924 tablet_replica.cc:442] T 26d6142f80d24395b9bee779d80aff2a P 403faea0cec54c4ba72e8997f14141f2: This TxnStatusTablet replica's current role is: FOLLOWER
I20240507 02:07:56.307029 7925 tablet_replica.cc:439] T 26d6142f80d24395b9bee779d80aff2a P b442a50dc84d4fb0bc141f691a5e8184: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b442a50dc84d4fb0bc141f691a5e8184" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35717 } } peers { permanent_uuid: "11db10e937ef426c9344e815d3e7262c" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44619 } } peers { permanent_uuid: "403faea0cec54c4ba72e8997f14141f2" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36381 } } }
I20240507 02:07:56.307076 7925 tablet_replica.cc:442] T 26d6142f80d24395b9bee779d80aff2a P b442a50dc84d4fb0bc141f691a5e8184: This TxnStatusTablet replica's current role is: FOLLOWER
I20240507 02:07:56.319921 7924 raft_consensus.cc:482] T 26d6142f80d24395b9bee779d80aff2a P 403faea0cec54c4ba72e8997f14141f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 02:07:56.319990 7924 raft_consensus.cc:504] T 26d6142f80d24395b9bee779d80aff2a P 403faea0cec54c4ba72e8997f14141f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b442a50dc84d4fb0bc141f691a5e8184" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35717 } } peers { permanent_uuid: "11db10e937ef426c9344e815d3e7262c" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44619 } } peers { permanent_uuid: "403faea0cec54c4ba72e8997f14141f2" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36381 } }
I20240507 02:07:56.320115 7924 leader_election.cc:288] T 26d6142f80d24395b9bee779d80aff2a P 403faea0cec54c4ba72e8997f14141f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b442a50dc84d4fb0bc141f691a5e8184 (127.31.131.65:35717), 11db10e937ef426c9344e815d3e7262c (127.31.131.66:44619)
I20240507 02:07:56.320291 7543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26d6142f80d24395b9bee779d80aff2a" candidate_uuid: "403faea0cec54c4ba72e8997f14141f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b442a50dc84d4fb0bc141f691a5e8184" is_pre_election: true
I20240507 02:07:56.320312 7702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26d6142f80d24395b9bee779d80aff2a" candidate_uuid: "403faea0cec54c4ba72e8997f14141f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11db10e937ef426c9344e815d3e7262c" is_pre_election: true
I20240507 02:07:56.320374 7702 raft_consensus.cc:2454] T 26d6142f80d24395b9bee779d80aff2a P 11db10e937ef426c9344e815d3e7262c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 403faea0cec54c4ba72e8997f14141f2 in term 0.
I20240507 02:07:56.320376 7543 raft_consensus.cc:2454] T 26d6142f80d24395b9bee779d80aff2a P b442a50dc84d4fb0bc141f691a5e8184 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 403faea0cec54c4ba72e8997f14141f2 in term 0.
I20240507 02:07:56.320510 7778 leader_election.cc:302] T 26d6142f80d24395b9bee779d80aff2a P 403faea0cec54c4ba72e8997f14141f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 403faea0cec54c4ba72e8997f14141f2, b442a50dc84d4fb0bc141f691a5e8184; no voters:
I20240507 02:07:56.320628 7924 raft_consensus.cc:2781] T 26d6142f80d24395b9bee779d80aff2a P 403faea0cec54c4ba72e8997f14141f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 02:07:56.320698 7924 raft_consensus.cc:482] T 26d6142f80d24395b9bee779d80aff2a P 403faea0cec54c4ba72e8997f14141f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 02:07:56.320711 7924 raft_consensus.cc:3037] T 26d6142f80d24395b9bee779d80aff2a P 403faea0cec54c4ba72e8997f14141f2 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:07:56.321329 7924 raft_consensus.cc:504] T 26d6142f80d24395b9bee779d80aff2a P 403faea0cec54c4ba72e8997f14141f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b442a50dc84d4fb0bc141f691a5e8184" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35717 } } peers { permanent_uuid: "11db10e937ef426c9344e815d3e7262c" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44619 } } peers { permanent_uuid: "403faea0cec54c4ba72e8997f14141f2" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36381 } }
I20240507 02:07:56.321441 7924 leader_election.cc:288] T 26d6142f80d24395b9bee779d80aff2a P 403faea0cec54c4ba72e8997f14141f2 [CANDIDATE]: Term 1 election: Requested vote from peers b442a50dc84d4fb0bc141f691a5e8184 (127.31.131.65:35717), 11db10e937ef426c9344e815d3e7262c (127.31.131.66:44619)
I20240507 02:07:56.321615 7702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26d6142f80d24395b9bee779d80aff2a" candidate_uuid: "403faea0cec54c4ba72e8997f14141f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11db10e937ef426c9344e815d3e7262c"
I20240507 02:07:56.321597 7543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26d6142f80d24395b9bee779d80aff2a" candidate_uuid: "403faea0cec54c4ba72e8997f14141f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b442a50dc84d4fb0bc141f691a5e8184"
I20240507 02:07:56.321686 7702 raft_consensus.cc:3037] T 26d6142f80d24395b9bee779d80aff2a P 11db10e937ef426c9344e815d3e7262c [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:07:56.321687 7543 raft_consensus.cc:3037] T 26d6142f80d24395b9bee779d80aff2a P b442a50dc84d4fb0bc141f691a5e8184 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:07:56.322381 7543 raft_consensus.cc:2454] T 26d6142f80d24395b9bee779d80aff2a P b442a50dc84d4fb0bc141f691a5e8184 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 403faea0cec54c4ba72e8997f14141f2 in term 1.
I20240507 02:07:56.322417 7702 raft_consensus.cc:2454] T 26d6142f80d24395b9bee779d80aff2a P 11db10e937ef426c9344e815d3e7262c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 403faea0cec54c4ba72e8997f14141f2 in term 1.
I20240507 02:07:56.322590 7778 leader_election.cc:302] T 26d6142f80d24395b9bee779d80aff2a P 403faea0cec54c4ba72e8997f14141f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 403faea0cec54c4ba72e8997f14141f2, b442a50dc84d4fb0bc141f691a5e8184; no voters:
I20240507 02:07:56.322680 7924 raft_consensus.cc:2781] T 26d6142f80d24395b9bee779d80aff2a P 403faea0cec54c4ba72e8997f14141f2 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:07:56.322808 7924 raft_consensus.cc:686] T 26d6142f80d24395b9bee779d80aff2a P 403faea0cec54c4ba72e8997f14141f2 [term 1 LEADER]: Becoming Leader. State: Replica: 403faea0cec54c4ba72e8997f14141f2, State: Running, Role: LEADER
I20240507 02:07:56.322870 7924 consensus_queue.cc:237] T 26d6142f80d24395b9bee779d80aff2a P 403faea0cec54c4ba72e8997f14141f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b442a50dc84d4fb0bc141f691a5e8184" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35717 } } peers { permanent_uuid: "11db10e937ef426c9344e815d3e7262c" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44619 } } peers { permanent_uuid: "403faea0cec54c4ba72e8997f14141f2" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36381 } }
I20240507 02:07:56.323207 7962 tablet_replica.cc:439] T 26d6142f80d24395b9bee779d80aff2a P 403faea0cec54c4ba72e8997f14141f2: TxnStatusTablet state changed. Reason: New leader 403faea0cec54c4ba72e8997f14141f2. Latest consensus state: current_term: 1 leader_uuid: "403faea0cec54c4ba72e8997f14141f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b442a50dc84d4fb0bc141f691a5e8184" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35717 } } peers { permanent_uuid: "11db10e937ef426c9344e815d3e7262c" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44619 } } peers { permanent_uuid: "403faea0cec54c4ba72e8997f14141f2" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36381 } } }
I20240507 02:07:56.323318 7962 tablet_replica.cc:442] T 26d6142f80d24395b9bee779d80aff2a P 403faea0cec54c4ba72e8997f14141f2: This TxnStatusTablet replica's current role is: LEADER
I20240507 02:07:56.323565 7964 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20240507 02:07:56.323535 7403 catalog_manager.cc:5484] T 26d6142f80d24395b9bee779d80aff2a P 403faea0cec54c4ba72e8997f14141f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 403faea0cec54c4ba72e8997f14141f2 (127.31.131.67). New cstate: current_term: 1 leader_uuid: "403faea0cec54c4ba72e8997f14141f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b442a50dc84d4fb0bc141f691a5e8184" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11db10e937ef426c9344e815d3e7262c" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "403faea0cec54c4ba72e8997f14141f2" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36381 } health_report { overall_health: HEALTHY } } }
I20240507 02:07:56.323671 7964 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20240507 02:07:56.330910 7918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:56.328852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57066 (local address 127.31.131.67:36381)
0507 02:07:56.328906 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:56.328908 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:56.328914 (+ 6us) server_negotiation.cc:407] Connection header received
0507 02:07:56.328939 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:56.328940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:56.328954 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:56.329010 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0507 02:07:56.329309 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:56.329734 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:56.330488 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:56.330628 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:56.330656 (+ 28us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0507 02:07:56.330731 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0507 02:07:56.330783 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:56.330845 (+ 62us) server_negotiation.cc:299] Negotiation successful
0507 02:07:56.330865 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20240507 02:07:56.330960 7958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:56.328617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.67:36381 (local address 127.0.0.1:57066)
0507 02:07:56.328695 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:56.328714 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:56.328763 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:56.329035 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:56.329036 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:56.329037 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0507 02:07:56.329247 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:56.329249 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:56.329793 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:56.329794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:56.330423 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:56.330425 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:56.330496 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:56.330512 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0507 02:07:56.330788 (+ 276us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0507 02:07:56.330794 (+ 6us) client_negotiation.cc:769] Sending connection context
0507 02:07:56.330840 (+ 46us) client_negotiation.cc:240] Negotiation successful
0507 02:07:56.330885 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20240507 02:07:56.332007 7543 raft_consensus.cc:1261] T 26d6142f80d24395b9bee779d80aff2a P b442a50dc84d4fb0bc141f691a5e8184 [term 1 FOLLOWER]: Refusing update from remote peer 403faea0cec54c4ba72e8997f14141f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 02:07:56.332084 7702 raft_consensus.cc:1261] T 26d6142f80d24395b9bee779d80aff2a P 11db10e937ef426c9344e815d3e7262c [term 1 FOLLOWER]: Refusing update from remote peer 403faea0cec54c4ba72e8997f14141f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 02:07:56.332275 7962 consensus_queue.cc:1035] T 26d6142f80d24395b9bee779d80aff2a P 403faea0cec54c4ba72e8997f14141f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11db10e937ef426c9344e815d3e7262c" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 02:07:56.332408 7924 consensus_queue.cc:1035] T 26d6142f80d24395b9bee779d80aff2a P 403faea0cec54c4ba72e8997f14141f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b442a50dc84d4fb0bc141f691a5e8184" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 02:07:56.333039 7927 tablet_replica.cc:439] T 26d6142f80d24395b9bee779d80aff2a P 11db10e937ef426c9344e815d3e7262c: TxnStatusTablet state changed. Reason: New leader 403faea0cec54c4ba72e8997f14141f2. Latest consensus state: current_term: 1 leader_uuid: "403faea0cec54c4ba72e8997f14141f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b442a50dc84d4fb0bc141f691a5e8184" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35717 } } peers { permanent_uuid: "11db10e937ef426c9344e815d3e7262c" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44619 } } peers { permanent_uuid: "403faea0cec54c4ba72e8997f14141f2" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36381 } } }
I20240507 02:07:56.333053 7925 tablet_replica.cc:439] T 26d6142f80d24395b9bee779d80aff2a P b442a50dc84d4fb0bc141f691a5e8184: TxnStatusTablet state changed. Reason: New leader 403faea0cec54c4ba72e8997f14141f2. Latest consensus state: current_term: 1 leader_uuid: "403faea0cec54c4ba72e8997f14141f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b442a50dc84d4fb0bc141f691a5e8184" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35717 } } peers { permanent_uuid: "11db10e937ef426c9344e815d3e7262c" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44619 } } peers { permanent_uuid: "403faea0cec54c4ba72e8997f14141f2" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36381 } } }
I20240507 02:07:56.333117 7927 tablet_replica.cc:442] T 26d6142f80d24395b9bee779d80aff2a P 11db10e937ef426c9344e815d3e7262c: This TxnStatusTablet replica's current role is: FOLLOWER
I20240507 02:07:56.333125 7925 tablet_replica.cc:442] T 26d6142f80d24395b9bee779d80aff2a P b442a50dc84d4fb0bc141f691a5e8184: This TxnStatusTablet replica's current role is: FOLLOWER
I20240507 02:07:56.333838 7924 tablet_replica.cc:439] T 26d6142f80d24395b9bee779d80aff2a P 403faea0cec54c4ba72e8997f14141f2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "403faea0cec54c4ba72e8997f14141f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b442a50dc84d4fb0bc141f691a5e8184" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35717 } } peers { permanent_uuid: "11db10e937ef426c9344e815d3e7262c" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44619 } } peers { permanent_uuid: "403faea0cec54c4ba72e8997f14141f2" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36381 } } }
I20240507 02:07:56.333927 7924 tablet_replica.cc:442] T 26d6142f80d24395b9bee779d80aff2a P 403faea0cec54c4ba72e8997f14141f2: This TxnStatusTablet replica's current role is: LEADER
I20240507 02:07:56.334481 7927 tablet_replica.cc:439] T 26d6142f80d24395b9bee779d80aff2a P 11db10e937ef426c9344e815d3e7262c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "403faea0cec54c4ba72e8997f14141f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b442a50dc84d4fb0bc141f691a5e8184" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35717 } } peers { permanent_uuid: "11db10e937ef426c9344e815d3e7262c" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44619 } } peers { permanent_uuid: "403faea0cec54c4ba72e8997f14141f2" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36381 } } }
I20240507 02:07:56.334633 7927 tablet_replica.cc:442] T 26d6142f80d24395b9bee779d80aff2a P 11db10e937ef426c9344e815d3e7262c: This TxnStatusTablet replica's current role is: FOLLOWER
I20240507 02:07:56.337015 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:56.335283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51622 (local address 127.31.131.65:35717)
0507 02:07:56.335321 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:56.335323 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:56.335329 (+ 6us) server_negotiation.cc:407] Connection header received
0507 02:07:56.335365 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:56.335365 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:56.335380 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:56.335438 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0507 02:07:56.335653 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:56.336084 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:56.336590 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:56.336713 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:56.336748 (+ 35us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0507 02:07:56.336822 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0507 02:07:56.336881 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:56.336934 (+ 53us) server_negotiation.cc:299] Negotiation successful
0507 02:07:56.336956 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20240507 02:07:56.342826 7918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:56.340906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:38179 (local address 127.31.131.67:36381)
0507 02:07:56.340949 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:56.340951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:56.341076 (+ 125us) server_negotiation.cc:407] Connection header received
0507 02:07:56.341128 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:56.341129 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:56.341138 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:56.341178 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0507 02:07:56.341433 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:56.341861 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:56.342428 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:56.342595 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:56.342614 (+ 19us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0507 02:07:56.342633 (+ 19us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0507 02:07:56.342678 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:56.342744 (+ 66us) server_negotiation.cc:299] Negotiation successful
0507 02:07:56.342762 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20240507 02:07:56.342842 7971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:56.340858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.67:36381 (local address 127.31.131.65:38179)
0507 02:07:56.341011 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:56.341019 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:56.341076 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:56.341183 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:56.341184 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:56.341185 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0507 02:07:56.341362 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:56.341364 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:56.341925 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:56.341926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:56.342360 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:56.342362 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:56.342418 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:56.342436 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0507 02:07:56.342687 (+ 251us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0507 02:07:56.342690 (+ 3us) client_negotiation.cc:769] Sending connection context
0507 02:07:56.342728 (+ 38us) client_negotiation.cc:240] Negotiation successful
0507 02:07:56.342770 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":64,"threads_started":1}
I20240507 02:07:56.343839 7962 tablet_replica.cc:439] T 26d6142f80d24395b9bee779d80aff2a P 403faea0cec54c4ba72e8997f14141f2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "403faea0cec54c4ba72e8997f14141f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b442a50dc84d4fb0bc141f691a5e8184" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35717 } } peers { permanent_uuid: "11db10e937ef426c9344e815d3e7262c" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44619 } } peers { permanent_uuid: "403faea0cec54c4ba72e8997f14141f2" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36381 } } }
I20240507 02:07:56.343917 7962 tablet_replica.cc:442] T 26d6142f80d24395b9bee779d80aff2a P 403faea0cec54c4ba72e8997f14141f2: This TxnStatusTablet replica's current role is: LEADER
I20240507 02:07:56.344161 7925 tablet_replica.cc:439] T 26d6142f80d24395b9bee779d80aff2a P b442a50dc84d4fb0bc141f691a5e8184: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "403faea0cec54c4ba72e8997f14141f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b442a50dc84d4fb0bc141f691a5e8184" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35717 } } peers { permanent_uuid: "11db10e937ef426c9344e815d3e7262c" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44619 } } peers { permanent_uuid: "403faea0cec54c4ba72e8997f14141f2" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36381 } } }
I20240507 02:07:56.344237 7925 tablet_replica.cc:442] T 26d6142f80d24395b9bee779d80aff2a P b442a50dc84d4fb0bc141f691a5e8184: This TxnStatusTablet replica's current role is: FOLLOWER
I20240507 02:07:56.345028 7844 raft_consensus.cc:1261] T 1a8af0a6630f48348c55b83cf96426e9 P 403faea0cec54c4ba72e8997f14141f2 [term 1 FOLLOWER]: Refusing update from remote peer b442a50dc84d4fb0bc141f691a5e8184: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 02:07:56.345199 7702 raft_consensus.cc:3037] T 1a8af0a6630f48348c55b83cf96426e9 P 11db10e937ef426c9344e815d3e7262c [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:07:56.345785 7702 raft_consensus.cc:1261] T 1a8af0a6630f48348c55b83cf96426e9 P 11db10e937ef426c9344e815d3e7262c [term 1 FOLLOWER]: Refusing update from remote peer b442a50dc84d4fb0bc141f691a5e8184: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 02:07:56.346009 7925 consensus_queue.cc:1035] T 1a8af0a6630f48348c55b83cf96426e9 P b442a50dc84d4fb0bc141f691a5e8184 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11db10e937ef426c9344e815d3e7262c" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 02:07:56.346159 7951 consensus_queue.cc:1035] T 1a8af0a6630f48348c55b83cf96426e9 P b442a50dc84d4fb0bc141f691a5e8184 [LEADER]: Connected to new peer: Peer: permanent_uuid: "403faea0cec54c4ba72e8997f14141f2" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 02:07:56.347339 7932 mvcc.cc:205] Tried to move back new op lower bound from 7024835282307723264 to 7024835281816702976. Current Snapshot: MvccSnapshot[applied={T|T < 7024835282307723264}]
I20240507 02:07:56.348008 7934 mvcc.cc:205] Tried to move back new op lower bound from 7024835282307723264 to 7024835281816702976. Current Snapshot: MvccSnapshot[applied={T|T < 7024835282307723264}]
I20240507 02:07:56.353852 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:56.351678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.65:35717 (local address 127.31.131.67:33743)
0507 02:07:56.351735 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:56.351746 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:56.351785 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:56.351910 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:56.351910 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:56.351911 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0507 02:07:56.352056 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:56.352058 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:56.352880 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:56.352881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:56.353368 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:56.353370 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:56.353424 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:56.353441 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0507 02:07:56.353709 (+ 268us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0507 02:07:56.353711 (+ 2us) client_negotiation.cc:769] Sending connection context
0507 02:07:56.353750 (+ 39us) client_negotiation.cc:240] Negotiation successful
0507 02:07:56.353788 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16}
I20240507 02:07:56.354077 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:56.351737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.67:33743 (local address 127.31.131.65:35717)
0507 02:07:56.351777 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:56.351779 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:56.351788 (+ 9us) server_negotiation.cc:407] Connection header received
0507 02:07:56.351849 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:56.351850 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:56.351859 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:56.351910 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0507 02:07:56.352098 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:56.352820 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:56.353432 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:56.353595 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:56.353614 (+ 19us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0507 02:07:56.353637 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0507 02:07:56.353678 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:56.354001 (+ 323us) server_negotiation.cc:299] Negotiation successful
0507 02:07:56.354024 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20240507 02:07:56.360427 7402 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46408:
table { table_name: "test-table" } modify_external_catalogs: true
I20240507 02:07:56.360512 7402 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46408:
table { table_name: "test-table" } modify_external_catalogs: true
I20240507 02:07:56.361069 7402 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5bf6e9ec74044a588b71bc5772a5d06d: Sending DeleteTablet for 3 replicas of tablet 1a8af0a6630f48348c55b83cf96426e9
I20240507 02:07:56.361291 7531 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a8af0a6630f48348c55b83cf96426e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:07:56 UTC) from {username='oryx', principal='oryx/127.31.131.126@KRBTEST.COM'} at 127.0.0.1:51620
I20240507 02:07:56.361330 7824 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a8af0a6630f48348c55b83cf96426e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:07:56 UTC) from {username='oryx', principal='oryx/127.31.131.126@KRBTEST.COM'} at 127.0.0.1:57052
I20240507 02:07:56.361377 7942 tablet_replica.cc:330] T 1a8af0a6630f48348c55b83cf96426e9 P b442a50dc84d4fb0bc141f691a5e8184: stopping tablet replica
I20240507 02:07:56.361392 7941 tablet_replica.cc:330] T 1a8af0a6630f48348c55b83cf96426e9 P 403faea0cec54c4ba72e8997f14141f2: stopping tablet replica
I20240507 02:07:56.361414 7942 raft_consensus.cc:2229] T 1a8af0a6630f48348c55b83cf96426e9 P b442a50dc84d4fb0bc141f691a5e8184 [term 1 LEADER]: Raft consensus shutting down.
I20240507 02:07:56.361424 7941 raft_consensus.cc:2229] T 1a8af0a6630f48348c55b83cf96426e9 P 403faea0cec54c4ba72e8997f14141f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 02:07:56.361451 7941 raft_consensus.cc:2258] T 1a8af0a6630f48348c55b83cf96426e9 P 403faea0cec54c4ba72e8997f14141f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 02:07:56.361471 7942 raft_consensus.cc:2258] T 1a8af0a6630f48348c55b83cf96426e9 P b442a50dc84d4fb0bc141f691a5e8184 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 02:07:56.361716 7683 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a8af0a6630f48348c55b83cf96426e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:07:56 UTC) from {username='oryx', principal='oryx/127.31.131.126@KRBTEST.COM'} at 127.0.0.1:56860
I20240507 02:07:56.361794 7943 tablet_replica.cc:330] T 1a8af0a6630f48348c55b83cf96426e9 P 11db10e937ef426c9344e815d3e7262c: stopping tablet replica
I20240507 02:07:56.361806 7942 ts_tablet_manager.cc:1875] T 1a8af0a6630f48348c55b83cf96426e9 P b442a50dc84d4fb0bc141f691a5e8184: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 02:07:56.361830 7943 raft_consensus.cc:2229] T 1a8af0a6630f48348c55b83cf96426e9 P 11db10e937ef426c9344e815d3e7262c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 02:07:56.361860 7943 raft_consensus.cc:2258] T 1a8af0a6630f48348c55b83cf96426e9 P 11db10e937ef426c9344e815d3e7262c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 02:07:56.361924 7941 ts_tablet_manager.cc:1875] T 1a8af0a6630f48348c55b83cf96426e9 P 403faea0cec54c4ba72e8997f14141f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 02:07:56.363310 7942 ts_tablet_manager.cc:1888] T 1a8af0a6630f48348c55b83cf96426e9 P b442a50dc84d4fb0bc141f691a5e8184: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240507 02:07:56.363366 7942 log.cc:1196] T 1a8af0a6630f48348c55b83cf96426e9 P b442a50dc84d4fb0bc141f691a5e8184: Deleting WAL directory at /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/ts-0/wal/wals/1a8af0a6630f48348c55b83cf96426e9
I20240507 02:07:56.363365 7941 ts_tablet_manager.cc:1888] T 1a8af0a6630f48348c55b83cf96426e9 P 403faea0cec54c4ba72e8997f14141f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240507 02:07:56.363408 7941 log.cc:1196] T 1a8af0a6630f48348c55b83cf96426e9 P 403faea0cec54c4ba72e8997f14141f2: Deleting WAL directory at /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/ts-2/wal/wals/1a8af0a6630f48348c55b83cf96426e9
I20240507 02:07:56.363564 7942 ts_tablet_manager.cc:1909] T 1a8af0a6630f48348c55b83cf96426e9 P b442a50dc84d4fb0bc141f691a5e8184: Deleting consensus metadata
I20240507 02:07:56.363590 7941 ts_tablet_manager.cc:1909] T 1a8af0a6630f48348c55b83cf96426e9 P 403faea0cec54c4ba72e8997f14141f2: Deleting consensus metadata
I20240507 02:07:56.363924 7390 catalog_manager.cc:4830] TS b442a50dc84d4fb0bc141f691a5e8184 (127.31.131.65:35717): tablet 1a8af0a6630f48348c55b83cf96426e9 (table test-table [id=f5e9a56872b1400a9db999b210c77620]) successfully deleted
I20240507 02:07:56.363997 7390 catalog_manager.cc:4830] TS 403faea0cec54c4ba72e8997f14141f2 (127.31.131.67:36381): tablet 1a8af0a6630f48348c55b83cf96426e9 (table test-table [id=f5e9a56872b1400a9db999b210c77620]) successfully deleted
I20240507 02:07:56.365082 7943 ts_tablet_manager.cc:1875] T 1a8af0a6630f48348c55b83cf96426e9 P 11db10e937ef426c9344e815d3e7262c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 02:07:56.366061 7943 ts_tablet_manager.cc:1888] T 1a8af0a6630f48348c55b83cf96426e9 P 11db10e937ef426c9344e815d3e7262c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240507 02:07:56.366111 7943 log.cc:1196] T 1a8af0a6630f48348c55b83cf96426e9 P 11db10e937ef426c9344e815d3e7262c: Deleting WAL directory at /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/ts-1/wal/wals/1a8af0a6630f48348c55b83cf96426e9
I20240507 02:07:56.366264 7943 ts_tablet_manager.cc:1909] T 1a8af0a6630f48348c55b83cf96426e9 P 11db10e937ef426c9344e815d3e7262c: Deleting consensus metadata
I20240507 02:07:56.366580 7389 catalog_manager.cc:4830] TS 11db10e937ef426c9344e815d3e7262c (127.31.131.66:44619): tablet 1a8af0a6630f48348c55b83cf96426e9 (table test-table [id=f5e9a56872b1400a9db999b210c77620]) successfully deleted
May 07 02:07:56 dist-test-slave-v0bg krb5kdc[7358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047676, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
May 07 02:07:56 dist-test-slave-v0bg krb5kdc[7358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
May 07 02:07:56 dist-test-slave-v0bg krb5kdc[7358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
W20240507 02:07:56.384618 7454 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:07:56.380008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46428 (local address 127.31.131.126:43871)
0507 02:07:56.380054 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:56.380057 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:56.380301 (+ 244us) server_negotiation.cc:407] Connection header received
0507 02:07:56.380331 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:56.380332 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:56.380342 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:56.380389 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:56.381018 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:56.381491 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:56.382042 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:56.382495 (+ 453us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:56.384566 (+ 2071us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46428: BlockingRecv error: recv got EOF from 127.0.0.1:46428 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
May 07 02:07:56 dist-test-slave-v0bg krb5kdc[7358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047676, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM
I20240507 02:07:56.391383 7454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:56.385533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46432 (local address 127.31.131.126:43871)
0507 02:07:56.385572 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:56.385574 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:56.385787 (+ 213us) server_negotiation.cc:407] Connection header received
0507 02:07:56.385842 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:56.385842 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:56.385852 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:56.385894 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:56.386390 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:56.386874 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:56.387346 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:56.387492 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:56.389178 (+ 1686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:56.389181 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:56.389181 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:07:56.389187 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:56.390427 (+ 1240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:56.390736 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:56.390737 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:56.390737 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:56.390763 (+ 26us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:56.390921 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:56.390922 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:56.390923 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:56.391001 (+ 78us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:56.391055 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:56.391211 (+ 156us) server_negotiation.cc:299] Negotiation successful
0507 02:07:56.391289 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20240507 02:07:56.392870 7402 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46432:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 02:07:56.393028 7402 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 02:07:56.394330 7531 tablet_service.cc:1467] Processing CreateTablet for tablet 6abdae735c7d427d967c3671332d58a4 (DEFAULT_TABLE table=test-table [id=ea01294fdbf1468f8dfc92ecc49dfd96]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:07:56.394359 7824 tablet_service.cc:1467] Processing CreateTablet for tablet 6abdae735c7d427d967c3671332d58a4 (DEFAULT_TABLE table=test-table [id=ea01294fdbf1468f8dfc92ecc49dfd96]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:07:56.394460 7531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6abdae735c7d427d967c3671332d58a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:07:56.394474 7824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6abdae735c7d427d967c3671332d58a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:07:56.394636 7683 tablet_service.cc:1467] Processing CreateTablet for tablet 6abdae735c7d427d967c3671332d58a4 (DEFAULT_TABLE table=test-table [id=ea01294fdbf1468f8dfc92ecc49dfd96]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:07:56.394737 7683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6abdae735c7d427d967c3671332d58a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:07:56.395750 7919 tablet_bootstrap.cc:492] T 6abdae735c7d427d967c3671332d58a4 P b442a50dc84d4fb0bc141f691a5e8184: Bootstrap starting.
I20240507 02:07:56.395889 7921 tablet_bootstrap.cc:492] T 6abdae735c7d427d967c3671332d58a4 P 11db10e937ef426c9344e815d3e7262c: Bootstrap starting.
I20240507 02:07:56.396123 7919 tablet_bootstrap.cc:654] T 6abdae735c7d427d967c3671332d58a4 P b442a50dc84d4fb0bc141f691a5e8184: Neither blocks nor log segments found. Creating new log.
I20240507 02:07:56.396147 7921 tablet_bootstrap.cc:654] T 6abdae735c7d427d967c3671332d58a4 P 11db10e937ef426c9344e815d3e7262c: Neither blocks nor log segments found. Creating new log.
I20240507 02:07:56.396970 7919 tablet_bootstrap.cc:492] T 6abdae735c7d427d967c3671332d58a4 P b442a50dc84d4fb0bc141f691a5e8184: No bootstrap required, opened a new log
I20240507 02:07:56.397017 7919 ts_tablet_manager.cc:1367] T 6abdae735c7d427d967c3671332d58a4 P b442a50dc84d4fb0bc141f691a5e8184: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240507 02:07:56.397158 7919 raft_consensus.cc:348] T 6abdae735c7d427d967c3671332d58a4 P b442a50dc84d4fb0bc141f691a5e8184 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b442a50dc84d4fb0bc141f691a5e8184" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35717 } } peers { permanent_uuid: "11db10e937ef426c9344e815d3e7262c" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44619 } } peers { permanent_uuid: "403faea0cec54c4ba72e8997f14141f2" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36381 } }
I20240507 02:07:56.397229 7919 raft_consensus.cc:374] T 6abdae735c7d427d967c3671332d58a4 P b442a50dc84d4fb0bc141f691a5e8184 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:07:56.397246 7919 raft_consensus.cc:729] T 6abdae735c7d427d967c3671332d58a4 P b442a50dc84d4fb0bc141f691a5e8184 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b442a50dc84d4fb0bc141f691a5e8184, State: Initialized, Role: FOLLOWER
I20240507 02:07:56.397287 7919 consensus_queue.cc:260] T 6abdae735c7d427d967c3671332d58a4 P b442a50dc84d4fb0bc141f691a5e8184 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b442a50dc84d4fb0bc141f691a5e8184" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35717 } } peers { permanent_uuid: "11db10e937ef426c9344e815d3e7262c" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44619 } } peers { permanent_uuid: "403faea0cec54c4ba72e8997f14141f2" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36381 } }
I20240507 02:07:56.397365 7919 ts_tablet_manager.cc:1398] T 6abdae735c7d427d967c3671332d58a4 P b442a50dc84d4fb0bc141f691a5e8184: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:56.397636 7920 tablet_bootstrap.cc:492] T 6abdae735c7d427d967c3671332d58a4 P 403faea0cec54c4ba72e8997f14141f2: Bootstrap starting.
I20240507 02:07:56.398037 7920 tablet_bootstrap.cc:654] T 6abdae735c7d427d967c3671332d58a4 P 403faea0cec54c4ba72e8997f14141f2: Neither blocks nor log segments found. Creating new log.
I20240507 02:07:56.398634 7920 tablet_bootstrap.cc:492] T 6abdae735c7d427d967c3671332d58a4 P 403faea0cec54c4ba72e8997f14141f2: No bootstrap required, opened a new log
I20240507 02:07:56.398682 7920 ts_tablet_manager.cc:1367] T 6abdae735c7d427d967c3671332d58a4 P 403faea0cec54c4ba72e8997f14141f2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240507 02:07:56.398821 7920 raft_consensus.cc:348] T 6abdae735c7d427d967c3671332d58a4 P 403faea0cec54c4ba72e8997f14141f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b442a50dc84d4fb0bc141f691a5e8184" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35717 } } peers { permanent_uuid: "11db10e937ef426c9344e815d3e7262c" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44619 } } peers { permanent_uuid: "403faea0cec54c4ba72e8997f14141f2" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36381 } }
I20240507 02:07:56.398885 7920 raft_consensus.cc:374] T 6abdae735c7d427d967c3671332d58a4 P 403faea0cec54c4ba72e8997f14141f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:07:56.398890 7921 tablet_bootstrap.cc:492] T 6abdae735c7d427d967c3671332d58a4 P 11db10e937ef426c9344e815d3e7262c: No bootstrap required, opened a new log
I20240507 02:07:56.398905 7920 raft_consensus.cc:729] T 6abdae735c7d427d967c3671332d58a4 P 403faea0cec54c4ba72e8997f14141f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 403faea0cec54c4ba72e8997f14141f2, State: Initialized, Role: FOLLOWER
I20240507 02:07:56.398927 7921 ts_tablet_manager.cc:1367] T 6abdae735c7d427d967c3671332d58a4 P 11db10e937ef426c9344e815d3e7262c: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.000s
I20240507 02:07:56.398942 7920 consensus_queue.cc:260] T 6abdae735c7d427d967c3671332d58a4 P 403faea0cec54c4ba72e8997f14141f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b442a50dc84d4fb0bc141f691a5e8184" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35717 } } peers { permanent_uuid: "11db10e937ef426c9344e815d3e7262c" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44619 } } peers { permanent_uuid: "403faea0cec54c4ba72e8997f14141f2" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36381 } }
I20240507 02:07:56.399025 7920 ts_tablet_manager.cc:1398] T 6abdae735c7d427d967c3671332d58a4 P 403faea0cec54c4ba72e8997f14141f2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:56.399024 7921 raft_consensus.cc:348] T 6abdae735c7d427d967c3671332d58a4 P 11db10e937ef426c9344e815d3e7262c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b442a50dc84d4fb0bc141f691a5e8184" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35717 } } peers { permanent_uuid: "11db10e937ef426c9344e815d3e7262c" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44619 } } peers { permanent_uuid: "403faea0cec54c4ba72e8997f14141f2" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36381 } }
I20240507 02:07:56.399080 7921 raft_consensus.cc:374] T 6abdae735c7d427d967c3671332d58a4 P 11db10e937ef426c9344e815d3e7262c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:07:56.399098 7921 raft_consensus.cc:729] T 6abdae735c7d427d967c3671332d58a4 P 11db10e937ef426c9344e815d3e7262c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11db10e937ef426c9344e815d3e7262c, State: Initialized, Role: FOLLOWER
I20240507 02:07:56.399137 7921 consensus_queue.cc:260] T 6abdae735c7d427d967c3671332d58a4 P 11db10e937ef426c9344e815d3e7262c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b442a50dc84d4fb0bc141f691a5e8184" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35717 } } peers { permanent_uuid: "11db10e937ef426c9344e815d3e7262c" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44619 } } peers { permanent_uuid: "403faea0cec54c4ba72e8997f14141f2" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36381 } }
I20240507 02:07:56.399226 7921 ts_tablet_manager.cc:1398] T 6abdae735c7d427d967c3671332d58a4 P 11db10e937ef426c9344e815d3e7262c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:56.594723 7962 raft_consensus.cc:482] T 6abdae735c7d427d967c3671332d58a4 P 403faea0cec54c4ba72e8997f14141f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 02:07:56.594844 7962 raft_consensus.cc:504] T 6abdae735c7d427d967c3671332d58a4 P 403faea0cec54c4ba72e8997f14141f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b442a50dc84d4fb0bc141f691a5e8184" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35717 } } peers { permanent_uuid: "11db10e937ef426c9344e815d3e7262c" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44619 } } peers { permanent_uuid: "403faea0cec54c4ba72e8997f14141f2" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36381 } }
I20240507 02:07:56.595036 7962 leader_election.cc:288] T 6abdae735c7d427d967c3671332d58a4 P 403faea0cec54c4ba72e8997f14141f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b442a50dc84d4fb0bc141f691a5e8184 (127.31.131.65:35717), 11db10e937ef426c9344e815d3e7262c (127.31.131.66:44619)
I20240507 02:07:56.595292 7702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6abdae735c7d427d967c3671332d58a4" candidate_uuid: "403faea0cec54c4ba72e8997f14141f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11db10e937ef426c9344e815d3e7262c" is_pre_election: true
I20240507 02:07:56.595280 7543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6abdae735c7d427d967c3671332d58a4" candidate_uuid: "403faea0cec54c4ba72e8997f14141f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b442a50dc84d4fb0bc141f691a5e8184" is_pre_election: true
I20240507 02:07:56.595391 7543 raft_consensus.cc:2454] T 6abdae735c7d427d967c3671332d58a4 P b442a50dc84d4fb0bc141f691a5e8184 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 403faea0cec54c4ba72e8997f14141f2 in term 0.
I20240507 02:07:56.595391 7702 raft_consensus.cc:2454] T 6abdae735c7d427d967c3671332d58a4 P 11db10e937ef426c9344e815d3e7262c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 403faea0cec54c4ba72e8997f14141f2 in term 0.
I20240507 02:07:56.595558 7778 leader_election.cc:302] T 6abdae735c7d427d967c3671332d58a4 P 403faea0cec54c4ba72e8997f14141f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 403faea0cec54c4ba72e8997f14141f2, b442a50dc84d4fb0bc141f691a5e8184; no voters:
I20240507 02:07:56.595669 7962 raft_consensus.cc:2781] T 6abdae735c7d427d967c3671332d58a4 P 403faea0cec54c4ba72e8997f14141f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 02:07:56.595702 7962 raft_consensus.cc:482] T 6abdae735c7d427d967c3671332d58a4 P 403faea0cec54c4ba72e8997f14141f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 02:07:56.595715 7962 raft_consensus.cc:3037] T 6abdae735c7d427d967c3671332d58a4 P 403faea0cec54c4ba72e8997f14141f2 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:07:56.596372 7962 raft_consensus.cc:504] T 6abdae735c7d427d967c3671332d58a4 P 403faea0cec54c4ba72e8997f14141f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b442a50dc84d4fb0bc141f691a5e8184" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35717 } } peers { permanent_uuid: "11db10e937ef426c9344e815d3e7262c" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44619 } } peers { permanent_uuid: "403faea0cec54c4ba72e8997f14141f2" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36381 } }
I20240507 02:07:56.596483 7962 leader_election.cc:288] T 6abdae735c7d427d967c3671332d58a4 P 403faea0cec54c4ba72e8997f14141f2 [CANDIDATE]: Term 1 election: Requested vote from peers b442a50dc84d4fb0bc141f691a5e8184 (127.31.131.65:35717), 11db10e937ef426c9344e815d3e7262c (127.31.131.66:44619)
I20240507 02:07:56.596638 7543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6abdae735c7d427d967c3671332d58a4" candidate_uuid: "403faea0cec54c4ba72e8997f14141f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b442a50dc84d4fb0bc141f691a5e8184"
I20240507 02:07:56.596669 7702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6abdae735c7d427d967c3671332d58a4" candidate_uuid: "403faea0cec54c4ba72e8997f14141f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11db10e937ef426c9344e815d3e7262c"
I20240507 02:07:56.596711 7543 raft_consensus.cc:3037] T 6abdae735c7d427d967c3671332d58a4 P b442a50dc84d4fb0bc141f691a5e8184 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:07:56.596751 7702 raft_consensus.cc:3037] T 6abdae735c7d427d967c3671332d58a4 P 11db10e937ef426c9344e815d3e7262c [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:07:56.597283 7543 raft_consensus.cc:2454] T 6abdae735c7d427d967c3671332d58a4 P b442a50dc84d4fb0bc141f691a5e8184 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 403faea0cec54c4ba72e8997f14141f2 in term 1.
I20240507 02:07:56.597407 7702 raft_consensus.cc:2454] T 6abdae735c7d427d967c3671332d58a4 P 11db10e937ef426c9344e815d3e7262c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 403faea0cec54c4ba72e8997f14141f2 in term 1.
I20240507 02:07:56.597481 7778 leader_election.cc:302] T 6abdae735c7d427d967c3671332d58a4 P 403faea0cec54c4ba72e8997f14141f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 403faea0cec54c4ba72e8997f14141f2, b442a50dc84d4fb0bc141f691a5e8184; no voters:
I20240507 02:07:56.597566 7962 raft_consensus.cc:2781] T 6abdae735c7d427d967c3671332d58a4 P 403faea0cec54c4ba72e8997f14141f2 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:07:56.597601 7962 raft_consensus.cc:686] T 6abdae735c7d427d967c3671332d58a4 P 403faea0cec54c4ba72e8997f14141f2 [term 1 LEADER]: Becoming Leader. State: Replica: 403faea0cec54c4ba72e8997f14141f2, State: Running, Role: LEADER
I20240507 02:07:56.597652 7962 consensus_queue.cc:237] T 6abdae735c7d427d967c3671332d58a4 P 403faea0cec54c4ba72e8997f14141f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b442a50dc84d4fb0bc141f691a5e8184" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35717 } } peers { permanent_uuid: "11db10e937ef426c9344e815d3e7262c" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44619 } } peers { permanent_uuid: "403faea0cec54c4ba72e8997f14141f2" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36381 } }
I20240507 02:07:56.598248 7403 catalog_manager.cc:5484] T 6abdae735c7d427d967c3671332d58a4 P 403faea0cec54c4ba72e8997f14141f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 403faea0cec54c4ba72e8997f14141f2 (127.31.131.67). New cstate: current_term: 1 leader_uuid: "403faea0cec54c4ba72e8997f14141f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b442a50dc84d4fb0bc141f691a5e8184" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11db10e937ef426c9344e815d3e7262c" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "403faea0cec54c4ba72e8997f14141f2" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36381 } health_report { overall_health: HEALTHY } } }
I20240507 02:07:56.626133 7918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:56.624375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57068 (local address 127.31.131.67:36381)
0507 02:07:56.624433 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:56.624435 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:56.624442 (+ 7us) server_negotiation.cc:407] Connection header received
0507 02:07:56.624466 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:56.624466 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:56.624476 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:56.624520 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0507 02:07:56.624743 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:56.625195 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:56.625738 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:56.625889 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:56.625905 (+ 16us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0507 02:07:56.625927 (+ 22us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0507 02:07:56.625969 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:56.626039 (+ 70us) server_negotiation.cc:299] Negotiation successful
0507 02:07:56.626061 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20240507 02:07:56.626987 7543 raft_consensus.cc:1261] T 6abdae735c7d427d967c3671332d58a4 P b442a50dc84d4fb0bc141f691a5e8184 [term 1 FOLLOWER]: Refusing update from remote peer 403faea0cec54c4ba72e8997f14141f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 02:07:56.627017 7702 raft_consensus.cc:1261] T 6abdae735c7d427d967c3671332d58a4 P 11db10e937ef426c9344e815d3e7262c [term 1 FOLLOWER]: Refusing update from remote peer 403faea0cec54c4ba72e8997f14141f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 02:07:56.627249 7962 consensus_queue.cc:1035] T 6abdae735c7d427d967c3671332d58a4 P 403faea0cec54c4ba72e8997f14141f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b442a50dc84d4fb0bc141f691a5e8184" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 02:07:56.627372 7924 consensus_queue.cc:1035] T 6abdae735c7d427d967c3671332d58a4 P 403faea0cec54c4ba72e8997f14141f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11db10e937ef426c9344e815d3e7262c" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 02:07:56.630712 7402 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46432:
table { table_name: "test-table" } modify_external_catalogs: true
I20240507 02:07:56.630817 7402 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46432:
table { table_name: "test-table" } modify_external_catalogs: true
I20240507 02:07:56.631448 7402 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5bf6e9ec74044a588b71bc5772a5d06d: Sending DeleteTablet for 3 replicas of tablet 6abdae735c7d427d967c3671332d58a4
I20240507 02:07:56.631728 7824 tablet_service.cc:1514] Processing DeleteTablet for tablet 6abdae735c7d427d967c3671332d58a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:07:56 UTC) from {username='oryx', principal='oryx/127.31.131.126@KRBTEST.COM'} at 127.0.0.1:57052
I20240507 02:07:56.631714 7531 tablet_service.cc:1514] Processing DeleteTablet for tablet 6abdae735c7d427d967c3671332d58a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:07:56 UTC) from {username='oryx', principal='oryx/127.31.131.126@KRBTEST.COM'} at 127.0.0.1:51620
I20240507 02:07:56.631819 7683 tablet_service.cc:1514] Processing DeleteTablet for tablet 6abdae735c7d427d967c3671332d58a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:07:56 UTC) from {username='oryx', principal='oryx/127.31.131.126@KRBTEST.COM'} at 127.0.0.1:56860
I20240507 02:07:56.631831 7942 tablet_replica.cc:330] T 6abdae735c7d427d967c3671332d58a4 P b442a50dc84d4fb0bc141f691a5e8184: stopping tablet replica
I20240507 02:07:56.631887 7943 tablet_replica.cc:330] T 6abdae735c7d427d967c3671332d58a4 P 11db10e937ef426c9344e815d3e7262c: stopping tablet replica
I20240507 02:07:56.631893 7942 raft_consensus.cc:2229] T 6abdae735c7d427d967c3671332d58a4 P b442a50dc84d4fb0bc141f691a5e8184 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 02:07:56.631935 7943 raft_consensus.cc:2229] T 6abdae735c7d427d967c3671332d58a4 P 11db10e937ef426c9344e815d3e7262c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 02:07:56.631937 7942 raft_consensus.cc:2258] T 6abdae735c7d427d967c3671332d58a4 P b442a50dc84d4fb0bc141f691a5e8184 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 02:07:56.631970 7943 raft_consensus.cc:2258] T 6abdae735c7d427d967c3671332d58a4 P 11db10e937ef426c9344e815d3e7262c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 02:07:56.632010 7941 tablet_replica.cc:330] T 6abdae735c7d427d967c3671332d58a4 P 403faea0cec54c4ba72e8997f14141f2: stopping tablet replica
I20240507 02:07:56.632059 7941 raft_consensus.cc:2229] T 6abdae735c7d427d967c3671332d58a4 P 403faea0cec54c4ba72e8997f14141f2 [term 1 LEADER]: Raft consensus shutting down.
I20240507 02:07:56.632133 7941 raft_consensus.cc:2258] T 6abdae735c7d427d967c3671332d58a4 P 403faea0cec54c4ba72e8997f14141f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 02:07:56.632246 7943 ts_tablet_manager.cc:1875] T 6abdae735c7d427d967c3671332d58a4 P 11db10e937ef426c9344e815d3e7262c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 02:07:56.632390 7402 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46432:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 02:07:56.632474 7941 ts_tablet_manager.cc:1875] T 6abdae735c7d427d967c3671332d58a4 P 403faea0cec54c4ba72e8997f14141f2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240507 02:07:56.632512 7402 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 02:07:56.633778 7683 tablet_service.cc:1467] Processing CreateTablet for tablet 2d5f8c7fdfc24214aa60363dbc52d128 (DEFAULT_TABLE table=test-table [id=ec602cd94e394d6fbbd42141ae9e6458]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:07:56.633810 7943 ts_tablet_manager.cc:1888] T 6abdae735c7d427d967c3671332d58a4 P 11db10e937ef426c9344e815d3e7262c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240507 02:07:56.633842 7941 ts_tablet_manager.cc:1888] T 6abdae735c7d427d967c3671332d58a4 P 403faea0cec54c4ba72e8997f14141f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240507 02:07:56.633867 7943 log.cc:1196] T 6abdae735c7d427d967c3671332d58a4 P 11db10e937ef426c9344e815d3e7262c: Deleting WAL directory at /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/ts-1/wal/wals/6abdae735c7d427d967c3671332d58a4
I20240507 02:07:56.633882 7683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d5f8c7fdfc24214aa60363dbc52d128. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:07:56.633884 7941 log.cc:1196] T 6abdae735c7d427d967c3671332d58a4 P 403faea0cec54c4ba72e8997f14141f2: Deleting WAL directory at /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/ts-2/wal/wals/6abdae735c7d427d967c3671332d58a4
I20240507 02:07:56.634035 7531 tablet_service.cc:1467] Processing CreateTablet for tablet 2d5f8c7fdfc24214aa60363dbc52d128 (DEFAULT_TABLE table=test-table [id=ec602cd94e394d6fbbd42141ae9e6458]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:07:56.634101 7943 ts_tablet_manager.cc:1909] T 6abdae735c7d427d967c3671332d58a4 P 11db10e937ef426c9344e815d3e7262c: Deleting consensus metadata
I20240507 02:07:56.634101 7941 ts_tablet_manager.cc:1909] T 6abdae735c7d427d967c3671332d58a4 P 403faea0cec54c4ba72e8997f14141f2: Deleting consensus metadata
I20240507 02:07:56.634121 7531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d5f8c7fdfc24214aa60363dbc52d128. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:07:56.634464 7389 catalog_manager.cc:4830] TS 11db10e937ef426c9344e815d3e7262c (127.31.131.66:44619): tablet 6abdae735c7d427d967c3671332d58a4 (table test-table [id=ea01294fdbf1468f8dfc92ecc49dfd96]) successfully deleted
I20240507 02:07:56.635001 7921 tablet_bootstrap.cc:492] T 2d5f8c7fdfc24214aa60363dbc52d128 P 11db10e937ef426c9344e815d3e7262c: Bootstrap starting.
I20240507 02:07:56.635244 7919 tablet_bootstrap.cc:492] T 2d5f8c7fdfc24214aa60363dbc52d128 P b442a50dc84d4fb0bc141f691a5e8184: Bootstrap starting.
I20240507 02:07:56.635341 7921 tablet_bootstrap.cc:654] T 2d5f8c7fdfc24214aa60363dbc52d128 P 11db10e937ef426c9344e815d3e7262c: Neither blocks nor log segments found. Creating new log.
I20240507 02:07:56.635367 7390 catalog_manager.cc:4830] TS 403faea0cec54c4ba72e8997f14141f2 (127.31.131.67:36381): tablet 6abdae735c7d427d967c3671332d58a4 (table test-table [id=ea01294fdbf1468f8dfc92ecc49dfd96]) successfully deleted
I20240507 02:07:56.635419 7824 tablet_service.cc:1467] Processing CreateTablet for tablet 2d5f8c7fdfc24214aa60363dbc52d128 (DEFAULT_TABLE table=test-table [id=ec602cd94e394d6fbbd42141ae9e6458]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:07:56.635512 7824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d5f8c7fdfc24214aa60363dbc52d128. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:07:56.635548 7919 tablet_bootstrap.cc:654] T 2d5f8c7fdfc24214aa60363dbc52d128 P b442a50dc84d4fb0bc141f691a5e8184: Neither blocks nor log segments found. Creating new log.
I20240507 02:07:56.635968 7942 ts_tablet_manager.cc:1875] T 6abdae735c7d427d967c3671332d58a4 P b442a50dc84d4fb0bc141f691a5e8184: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 02:07:56.635998 7921 tablet_bootstrap.cc:492] T 2d5f8c7fdfc24214aa60363dbc52d128 P 11db10e937ef426c9344e815d3e7262c: No bootstrap required, opened a new log
I20240507 02:07:56.636042 7921 ts_tablet_manager.cc:1367] T 2d5f8c7fdfc24214aa60363dbc52d128 P 11db10e937ef426c9344e815d3e7262c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240507 02:07:56.636184 7921 raft_consensus.cc:348] T 2d5f8c7fdfc24214aa60363dbc52d128 P 11db10e937ef426c9344e815d3e7262c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b442a50dc84d4fb0bc141f691a5e8184" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35717 } } peers { permanent_uuid: "11db10e937ef426c9344e815d3e7262c" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44619 } } peers { permanent_uuid: "403faea0cec54c4ba72e8997f14141f2" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36381 } }
I20240507 02:07:56.636245 7921 raft_consensus.cc:374] T 2d5f8c7fdfc24214aa60363dbc52d128 P 11db10e937ef426c9344e815d3e7262c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:07:56.636266 7921 raft_consensus.cc:729] T 2d5f8c7fdfc24214aa60363dbc52d128 P 11db10e937ef426c9344e815d3e7262c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11db10e937ef426c9344e815d3e7262c, State: Initialized, Role: FOLLOWER
I20240507 02:07:56.636320 7921 consensus_queue.cc:260] T 2d5f8c7fdfc24214aa60363dbc52d128 P 11db10e937ef426c9344e815d3e7262c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b442a50dc84d4fb0bc141f691a5e8184" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35717 } } peers { permanent_uuid: "11db10e937ef426c9344e815d3e7262c" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44619 } } peers { permanent_uuid: "403faea0cec54c4ba72e8997f14141f2" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36381 } }
I20240507 02:07:56.636407 7921 ts_tablet_manager.cc:1398] T 2d5f8c7fdfc24214aa60363dbc52d128 P 11db10e937ef426c9344e815d3e7262c: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20240507 02:07:56.636711 7920 tablet_bootstrap.cc:492] T 2d5f8c7fdfc24214aa60363dbc52d128 P 403faea0cec54c4ba72e8997f14141f2: Bootstrap starting.
I20240507 02:07:56.637002 7920 tablet_bootstrap.cc:654] T 2d5f8c7fdfc24214aa60363dbc52d128 P 403faea0cec54c4ba72e8997f14141f2: Neither blocks nor log segments found. Creating new log.
I20240507 02:07:56.637008 7942 ts_tablet_manager.cc:1888] T 6abdae735c7d427d967c3671332d58a4 P b442a50dc84d4fb0bc141f691a5e8184: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240507 02:07:56.637048 7942 log.cc:1196] T 6abdae735c7d427d967c3671332d58a4 P b442a50dc84d4fb0bc141f691a5e8184: Deleting WAL directory at /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/ts-0/wal/wals/6abdae735c7d427d967c3671332d58a4
I20240507 02:07:56.637236 7942 ts_tablet_manager.cc:1909] T 6abdae735c7d427d967c3671332d58a4 P b442a50dc84d4fb0bc141f691a5e8184: Deleting consensus metadata
I20240507 02:07:56.637609 7920 tablet_bootstrap.cc:492] T 2d5f8c7fdfc24214aa60363dbc52d128 P 403faea0cec54c4ba72e8997f14141f2: No bootstrap required, opened a new log
I20240507 02:07:56.637595 7390 catalog_manager.cc:4830] TS b442a50dc84d4fb0bc141f691a5e8184 (127.31.131.65:35717): tablet 6abdae735c7d427d967c3671332d58a4 (table test-table [id=ea01294fdbf1468f8dfc92ecc49dfd96]) successfully deleted
I20240507 02:07:56.637661 7920 ts_tablet_manager.cc:1367] T 2d5f8c7fdfc24214aa60363dbc52d128 P 403faea0cec54c4ba72e8997f14141f2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240507 02:07:56.637773 7920 raft_consensus.cc:348] T 2d5f8c7fdfc24214aa60363dbc52d128 P 403faea0cec54c4ba72e8997f14141f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b442a50dc84d4fb0bc141f691a5e8184" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35717 } } peers { permanent_uuid: "11db10e937ef426c9344e815d3e7262c" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44619 } } peers { permanent_uuid: "403faea0cec54c4ba72e8997f14141f2" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36381 } }
I20240507 02:07:56.637835 7920 raft_consensus.cc:374] T 2d5f8c7fdfc24214aa60363dbc52d128 P 403faea0cec54c4ba72e8997f14141f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:07:56.637859 7920 raft_consensus.cc:729] T 2d5f8c7fdfc24214aa60363dbc52d128 P 403faea0cec54c4ba72e8997f14141f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 403faea0cec54c4ba72e8997f14141f2, State: Initialized, Role: FOLLOWER
I20240507 02:07:56.637897 7920 consensus_queue.cc:260] T 2d5f8c7fdfc24214aa60363dbc52d128 P 403faea0cec54c4ba72e8997f14141f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b442a50dc84d4fb0bc141f691a5e8184" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35717 } } peers { permanent_uuid: "11db10e937ef426c9344e815d3e7262c" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44619 } } peers { permanent_uuid: "403faea0cec54c4ba72e8997f14141f2" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36381 } }
I20240507 02:07:56.637976 7919 tablet_bootstrap.cc:492] T 2d5f8c7fdfc24214aa60363dbc52d128 P b442a50dc84d4fb0bc141f691a5e8184: No bootstrap required, opened a new log
I20240507 02:07:56.637988 7920 ts_tablet_manager.cc:1398] T 2d5f8c7fdfc24214aa60363dbc52d128 P 403faea0cec54c4ba72e8997f14141f2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:56.638031 7919 ts_tablet_manager.cc:1367] T 2d5f8c7fdfc24214aa60363dbc52d128 P b442a50dc84d4fb0bc141f691a5e8184: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240507 02:07:56.638157 7919 raft_consensus.cc:348] T 2d5f8c7fdfc24214aa60363dbc52d128 P b442a50dc84d4fb0bc141f691a5e8184 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b442a50dc84d4fb0bc141f691a5e8184" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35717 } } peers { permanent_uuid: "11db10e937ef426c9344e815d3e7262c" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44619 } } peers { permanent_uuid: "403faea0cec54c4ba72e8997f14141f2" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36381 } }
I20240507 02:07:56.638226 7919 raft_consensus.cc:374] T 2d5f8c7fdfc24214aa60363dbc52d128 P b442a50dc84d4fb0bc141f691a5e8184 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:07:56.638247 7919 raft_consensus.cc:729] T 2d5f8c7fdfc24214aa60363dbc52d128 P b442a50dc84d4fb0bc141f691a5e8184 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b442a50dc84d4fb0bc141f691a5e8184, State: Initialized, Role: FOLLOWER
I20240507 02:07:56.638283 7919 consensus_queue.cc:260] T 2d5f8c7fdfc24214aa60363dbc52d128 P b442a50dc84d4fb0bc141f691a5e8184 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b442a50dc84d4fb0bc141f691a5e8184" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35717 } } peers { permanent_uuid: "11db10e937ef426c9344e815d3e7262c" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44619 } } peers { permanent_uuid: "403faea0cec54c4ba72e8997f14141f2" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36381 } }
I20240507 02:07:56.638367 7919 ts_tablet_manager.cc:1398] T 2d5f8c7fdfc24214aa60363dbc52d128 P b442a50dc84d4fb0bc141f691a5e8184: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20240507 02:07:56.771633 7924 raft_consensus.cc:482] T 2d5f8c7fdfc24214aa60363dbc52d128 P 403faea0cec54c4ba72e8997f14141f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 02:07:56.771752 7924 raft_consensus.cc:504] T 2d5f8c7fdfc24214aa60363dbc52d128 P 403faea0cec54c4ba72e8997f14141f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b442a50dc84d4fb0bc141f691a5e8184" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35717 } } peers { permanent_uuid: "11db10e937ef426c9344e815d3e7262c" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44619 } } peers { permanent_uuid: "403faea0cec54c4ba72e8997f14141f2" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36381 } }
I20240507 02:07:56.771908 7924 leader_election.cc:288] T 2d5f8c7fdfc24214aa60363dbc52d128 P 403faea0cec54c4ba72e8997f14141f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b442a50dc84d4fb0bc141f691a5e8184 (127.31.131.65:35717), 11db10e937ef426c9344e815d3e7262c (127.31.131.66:44619)
I20240507 02:07:56.772146 7702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d5f8c7fdfc24214aa60363dbc52d128" candidate_uuid: "403faea0cec54c4ba72e8997f14141f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11db10e937ef426c9344e815d3e7262c" is_pre_election: true
I20240507 02:07:56.772153 7543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d5f8c7fdfc24214aa60363dbc52d128" candidate_uuid: "403faea0cec54c4ba72e8997f14141f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b442a50dc84d4fb0bc141f691a5e8184" is_pre_election: true
I20240507 02:07:56.772244 7702 raft_consensus.cc:2454] T 2d5f8c7fdfc24214aa60363dbc52d128 P 11db10e937ef426c9344e815d3e7262c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 403faea0cec54c4ba72e8997f14141f2 in term 0.
I20240507 02:07:56.772250 7543 raft_consensus.cc:2454] T 2d5f8c7fdfc24214aa60363dbc52d128 P b442a50dc84d4fb0bc141f691a5e8184 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 403faea0cec54c4ba72e8997f14141f2 in term 0.
I20240507 02:07:56.772413 7778 leader_election.cc:302] T 2d5f8c7fdfc24214aa60363dbc52d128 P 403faea0cec54c4ba72e8997f14141f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 403faea0cec54c4ba72e8997f14141f2, b442a50dc84d4fb0bc141f691a5e8184; no voters:
I20240507 02:07:56.772511 7924 raft_consensus.cc:2781] T 2d5f8c7fdfc24214aa60363dbc52d128 P 403faea0cec54c4ba72e8997f14141f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 02:07:56.772549 7924 raft_consensus.cc:482] T 2d5f8c7fdfc24214aa60363dbc52d128 P 403faea0cec54c4ba72e8997f14141f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 02:07:56.772578 7924 raft_consensus.cc:3037] T 2d5f8c7fdfc24214aa60363dbc52d128 P 403faea0cec54c4ba72e8997f14141f2 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:07:56.773233 7924 raft_consensus.cc:504] T 2d5f8c7fdfc24214aa60363dbc52d128 P 403faea0cec54c4ba72e8997f14141f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b442a50dc84d4fb0bc141f691a5e8184" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35717 } } peers { permanent_uuid: "11db10e937ef426c9344e815d3e7262c" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44619 } } peers { permanent_uuid: "403faea0cec54c4ba72e8997f14141f2" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36381 } }
I20240507 02:07:56.773356 7924 leader_election.cc:288] T 2d5f8c7fdfc24214aa60363dbc52d128 P 403faea0cec54c4ba72e8997f14141f2 [CANDIDATE]: Term 1 election: Requested vote from peers b442a50dc84d4fb0bc141f691a5e8184 (127.31.131.65:35717), 11db10e937ef426c9344e815d3e7262c (127.31.131.66:44619)
I20240507 02:07:56.773515 7543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d5f8c7fdfc24214aa60363dbc52d128" candidate_uuid: "403faea0cec54c4ba72e8997f14141f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b442a50dc84d4fb0bc141f691a5e8184"
I20240507 02:07:56.773523 7702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d5f8c7fdfc24214aa60363dbc52d128" candidate_uuid: "403faea0cec54c4ba72e8997f14141f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11db10e937ef426c9344e815d3e7262c"
I20240507 02:07:56.773591 7543 raft_consensus.cc:3037] T 2d5f8c7fdfc24214aa60363dbc52d128 P b442a50dc84d4fb0bc141f691a5e8184 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:07:56.773591 7702 raft_consensus.cc:3037] T 2d5f8c7fdfc24214aa60363dbc52d128 P 11db10e937ef426c9344e815d3e7262c [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:07:56.774180 7702 raft_consensus.cc:2454] T 2d5f8c7fdfc24214aa60363dbc52d128 P 11db10e937ef426c9344e815d3e7262c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 403faea0cec54c4ba72e8997f14141f2 in term 1.
I20240507 02:07:56.774238 7543 raft_consensus.cc:2454] T 2d5f8c7fdfc24214aa60363dbc52d128 P b442a50dc84d4fb0bc141f691a5e8184 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 403faea0cec54c4ba72e8997f14141f2 in term 1.
I20240507 02:07:56.774379 7777 leader_election.cc:302] T 2d5f8c7fdfc24214aa60363dbc52d128 P 403faea0cec54c4ba72e8997f14141f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11db10e937ef426c9344e815d3e7262c, 403faea0cec54c4ba72e8997f14141f2; no voters:
I20240507 02:07:56.774461 7924 raft_consensus.cc:2781] T 2d5f8c7fdfc24214aa60363dbc52d128 P 403faea0cec54c4ba72e8997f14141f2 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:07:56.774525 7924 raft_consensus.cc:686] T 2d5f8c7fdfc24214aa60363dbc52d128 P 403faea0cec54c4ba72e8997f14141f2 [term 1 LEADER]: Becoming Leader. State: Replica: 403faea0cec54c4ba72e8997f14141f2, State: Running, Role: LEADER
I20240507 02:07:56.774596 7924 consensus_queue.cc:237] T 2d5f8c7fdfc24214aa60363dbc52d128 P 403faea0cec54c4ba72e8997f14141f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b442a50dc84d4fb0bc141f691a5e8184" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35717 } } peers { permanent_uuid: "11db10e937ef426c9344e815d3e7262c" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44619 } } peers { permanent_uuid: "403faea0cec54c4ba72e8997f14141f2" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36381 } }
I20240507 02:07:56.775135 7403 catalog_manager.cc:5484] T 2d5f8c7fdfc24214aa60363dbc52d128 P 403faea0cec54c4ba72e8997f14141f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 403faea0cec54c4ba72e8997f14141f2 (127.31.131.67). New cstate: current_term: 1 leader_uuid: "403faea0cec54c4ba72e8997f14141f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b442a50dc84d4fb0bc141f691a5e8184" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11db10e937ef426c9344e815d3e7262c" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "403faea0cec54c4ba72e8997f14141f2" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 36381 } health_report { overall_health: HEALTHY } } }
I20240507 02:07:56.786646 7918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:56.785005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.67:34825 (local address 127.31.131.67:36381)
0507 02:07:56.785040 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:56.785042 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:56.785048 (+ 6us) server_negotiation.cc:407] Connection header received
0507 02:07:56.785062 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:56.785063 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:56.785071 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:56.785112 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0507 02:07:56.785319 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:56.785754 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:56.786274 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:56.786396 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:56.786410 (+ 14us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0507 02:07:56.786429 (+ 19us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0507 02:07:56.786471 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:56.786535 (+ 64us) server_negotiation.cc:299] Negotiation successful
0507 02:07:56.786573 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20240507 02:07:56.786646 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:56.784834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.67:36381 (local address 127.31.131.67:34825)
0507 02:07:56.784887 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:56.784897 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:56.784928 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:56.785121 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:56.785122 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:56.785123 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0507 02:07:56.785263 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:56.785264 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:56.785815 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:56.785816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:56.786201 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:56.786202 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:56.786261 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:56.786267 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0507 02:07:56.786481 (+ 214us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0507 02:07:56.786482 (+ 1us) client_negotiation.cc:769] Sending connection context
0507 02:07:56.786523 (+ 41us) client_negotiation.cc:240] Negotiation successful
0507 02:07:56.786589 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20240507 02:07:56.788210 7543 raft_consensus.cc:1261] T 2d5f8c7fdfc24214aa60363dbc52d128 P b442a50dc84d4fb0bc141f691a5e8184 [term 1 FOLLOWER]: Refusing update from remote peer 403faea0cec54c4ba72e8997f14141f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 02:07:56.788209 7702 raft_consensus.cc:1261] T 2d5f8c7fdfc24214aa60363dbc52d128 P 11db10e937ef426c9344e815d3e7262c [term 1 FOLLOWER]: Refusing update from remote peer 403faea0cec54c4ba72e8997f14141f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 02:07:56.788399 7962 consensus_queue.cc:1035] T 2d5f8c7fdfc24214aa60363dbc52d128 P 403faea0cec54c4ba72e8997f14141f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b442a50dc84d4fb0bc141f691a5e8184" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 02:07:56.788486 7924 consensus_queue.cc:1035] T 2d5f8c7fdfc24214aa60363dbc52d128 P 403faea0cec54c4ba72e8997f14141f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11db10e937ef426c9344e815d3e7262c" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 02:07:56.797844 7401 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46432:
table { table_name: "test-table" } modify_external_catalogs: true
I20240507 02:07:56.797937 7401 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46432:
table { table_name: "test-table" } modify_external_catalogs: true
I20240507 02:07:56.798512 7401 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5bf6e9ec74044a588b71bc5772a5d06d: Sending DeleteTablet for 3 replicas of tablet 2d5f8c7fdfc24214aa60363dbc52d128
I20240507 02:07:56.798787 7531 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d5f8c7fdfc24214aa60363dbc52d128 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:07:56 UTC) from {username='oryx', principal='oryx/127.31.131.126@KRBTEST.COM'} at 127.0.0.1:51620
I20240507 02:07:56.798789 7824 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d5f8c7fdfc24214aa60363dbc52d128 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:07:56 UTC) from {username='oryx', principal='oryx/127.31.131.126@KRBTEST.COM'} at 127.0.0.1:57052
I20240507 02:07:56.798888 7942 tablet_replica.cc:330] T 2d5f8c7fdfc24214aa60363dbc52d128 P b442a50dc84d4fb0bc141f691a5e8184: stopping tablet replica
I20240507 02:07:56.798892 7683 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d5f8c7fdfc24214aa60363dbc52d128 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:07:56 UTC) from {username='oryx', principal='oryx/127.31.131.126@KRBTEST.COM'} at 127.0.0.1:56860
I20240507 02:07:56.798939 7942 raft_consensus.cc:2229] T 2d5f8c7fdfc24214aa60363dbc52d128 P b442a50dc84d4fb0bc141f691a5e8184 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 02:07:56.798966 7941 tablet_replica.cc:330] T 2d5f8c7fdfc24214aa60363dbc52d128 P 403faea0cec54c4ba72e8997f14141f2: stopping tablet replica
I20240507 02:07:56.798978 7942 raft_consensus.cc:2258] T 2d5f8c7fdfc24214aa60363dbc52d128 P b442a50dc84d4fb0bc141f691a5e8184 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 02:07:56.799022 7941 raft_consensus.cc:2229] T 2d5f8c7fdfc24214aa60363dbc52d128 P 403faea0cec54c4ba72e8997f14141f2 [term 1 LEADER]: Raft consensus shutting down.
I20240507 02:07:56.799082 7941 raft_consensus.cc:2258] T 2d5f8c7fdfc24214aa60363dbc52d128 P 403faea0cec54c4ba72e8997f14141f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 02:07:56.799146 7943 tablet_replica.cc:330] T 2d5f8c7fdfc24214aa60363dbc52d128 P 11db10e937ef426c9344e815d3e7262c: stopping tablet replica
I20240507 02:07:56.799202 7943 raft_consensus.cc:2229] T 2d5f8c7fdfc24214aa60363dbc52d128 P 11db10e937ef426c9344e815d3e7262c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 02:07:56.799237 7943 raft_consensus.cc:2258] T 2d5f8c7fdfc24214aa60363dbc52d128 P 11db10e937ef426c9344e815d3e7262c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 02:07:56.799261 7942 ts_tablet_manager.cc:1875] T 2d5f8c7fdfc24214aa60363dbc52d128 P b442a50dc84d4fb0bc141f691a5e8184: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 02:07:56.799397 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 7476
I20240507 02:07:56.799592 7943 ts_tablet_manager.cc:1875] T 2d5f8c7fdfc24214aa60363dbc52d128 P 11db10e937ef426c9344e815d3e7262c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 02:07:56.800127 7941 ts_tablet_manager.cc:1875] T 2d5f8c7fdfc24214aa60363dbc52d128 P 403faea0cec54c4ba72e8997f14141f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 02:07:56.801039 7943 ts_tablet_manager.cc:1888] T 2d5f8c7fdfc24214aa60363dbc52d128 P 11db10e937ef426c9344e815d3e7262c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240507 02:07:56.801087 7943 log.cc:1196] T 2d5f8c7fdfc24214aa60363dbc52d128 P 11db10e937ef426c9344e815d3e7262c: Deleting WAL directory at /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/ts-1/wal/wals/2d5f8c7fdfc24214aa60363dbc52d128
I20240507 02:07:56.801290 7943 ts_tablet_manager.cc:1909] T 2d5f8c7fdfc24214aa60363dbc52d128 P 11db10e937ef426c9344e815d3e7262c: Deleting consensus metadata
I20240507 02:07:56.801425 7941 ts_tablet_manager.cc:1888] T 2d5f8c7fdfc24214aa60363dbc52d128 P 403faea0cec54c4ba72e8997f14141f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240507 02:07:56.801470 7941 log.cc:1196] T 2d5f8c7fdfc24214aa60363dbc52d128 P 403faea0cec54c4ba72e8997f14141f2: Deleting WAL directory at /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1715047604443657-32269-0/minicluster-data/ts-2/wal/wals/2d5f8c7fdfc24214aa60363dbc52d128
I20240507 02:07:56.801645 7941 ts_tablet_manager.cc:1909] T 2d5f8c7fdfc24214aa60363dbc52d128 P 403faea0cec54c4ba72e8997f14141f2: Deleting consensus metadata
I20240507 02:07:56.801797 7389 catalog_manager.cc:4830] TS 11db10e937ef426c9344e815d3e7262c (127.31.131.66:44619): tablet 2d5f8c7fdfc24214aa60363dbc52d128 (table test-table [id=ec602cd94e394d6fbbd42141ae9e6458]) successfully deleted
I20240507 02:07:56.802007 7390 catalog_manager.cc:4830] TS 403faea0cec54c4ba72e8997f14141f2 (127.31.131.67:36381): tablet 2d5f8c7fdfc24214aa60363dbc52d128 (table test-table [id=ec602cd94e394d6fbbd42141ae9e6458]) successfully deleted
W20240507 02:07:56.805631 7390 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.131.65:35717 (error 108)
I20240507 02:07:56.805830 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 7616
I20240507 02:07:56.806360 8006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:56.805942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.65:35717 (local address 127.0.0.1:51626)
0507 02:07:56.806291 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:56.806324 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.65:35717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":36,"threads_started":1}
W20240507 02:07:56.806433 7390 catalog_manager.cc:4573] TS b442a50dc84d4fb0bc141f691a5e8184 (127.31.131.65:35717): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2d5f8c7fdfc24214aa60363dbc52d128: Network error: Client connection negotiation failed: client connection to 127.31.131.65:35717: connect: Connection refused (error 111)
I20240507 02:07:56.811836 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 7759
I20240507 02:07:56.816658 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 7371
2024-05-07T02:07:56Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2581 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1715047604443657-32269-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 07 02:07:56 dist-test-slave-v0bg krb5kdc[8010](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 07 02:07:56 dist-test-slave-v0bg krb5kdc[8010](info): set up 2 sockets
May 07 02:07:56 dist-test-slave-v0bg krb5kdc[8010](info): commencing operation
krb5kdc: starting...
W20240507 02:07:57.703311 32269 mini_kdc.cc:121] Time spent starting KDC: real 0.867s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1715047604443657-32269-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1715047604443657-32269-0/krb5kdc/test-user.keytab.
May 07 02:07:57 dist-test-slave-v0bg krb5kdc[8010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-07T02:07:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-07T02:07:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.131.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab.
Entry for principal kudu/127.31.131.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab.
I20240507 02:07:57.810746 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1715047604443657-32269-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1715047604443657-32269-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1715047604443657-32269-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:37725
--webserver_interface=127.31.131.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab
--principal=kudu/127.31.131.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:37505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.131.126:37725
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:07:57.886339 8023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:07:57.886516 8023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:07:57.886535 8023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:07:57.887805 8023 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240507 02:07:57.887843 8023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:07:57.887856 8023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:07:57.887866 8023 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240507 02:07:57.887876 8023 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240507 02:07:57.889400 8023 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:37505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1715047604443657-32269-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1715047604443657-32269-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.131.126:37725
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:37725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.131.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8023
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1715047604443657-32269-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:07:57.889568 8023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:07:57.889712 8023 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:07:57.893256 8031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:07:57.893318 8023 server_base.cc:1034] running on GCE node
W20240507 02:07:57.893368 8028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:07:57.893553 8029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:07:57.893759 8023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:07:57.894003 8023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:07:57.895124 8023 hybrid_clock.cc:648] HybridClock initialized: now 1715047677895118 us; error 25 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2024-05-07T02:07:57Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1081 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 07 02:07:57 dist-test-slave-v0bg krb5kdc[8036](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 07 02:07:57 dist-test-slave-v0bg krb5kdc[8036](info): set up 2 sockets
May 07 02:07:57 dist-test-slave-v0bg krb5kdc[8036](info): commencing operation
krb5kdc: starting...
W20240507 02:07:58.787420 32269 mini_kdc.cc:121] Time spent starting KDC: real 0.870s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/krb5kdc/test-user.keytab.
May 07 02:07:58 dist-test-slave-v0bg krb5kdc[8036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047678, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-07T02:07:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-07T02:07:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.131.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab.
Entry for principal kudu/127.31.131.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab.
I20240507 02:07:58.887017 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:39211
--webserver_interface=127.31.131.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab
--principal=kudu/127.31.131.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:39087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.131.126:39211
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:07:58.964299 8049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:07:58.964488 8049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:07:58.964509 8049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:07:58.965788 8049 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240507 02:07:58.965831 8049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:07:58.965842 8049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:07:58.965852 8049 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240507 02:07:58.965867 8049 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240507 02:07:58.967394 8049 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:39087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.131.126:39211
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab
--principal=kudu/127.31.131.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:39211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.131.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8049
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:07:58.967573 8049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:07:58.967720 8049 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:07:58.971385 8054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:07:58.971434 8049 server_base.cc:1034] running on GCE node
W20240507 02:07:58.971562 8057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:07:58.971987 8055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:07:58.972287 8049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:07:58.972548 8049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:07:58.973688 8049 hybrid_clock.cc:648] HybridClock initialized: now 1715047678973670 us; error 27 us; skew 500 ppm
May 07 02:07:58 dist-test-slave-v0bg krb5kdc[8036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047678, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:07:58.976642 8049 init.cc:377] Logged in from keytab as kudu/127.31.131.126@KRBTEST.COM (short username kudu)
I20240507 02:07:58.977124 8049 webserver.cc:460] Webserver started at http://127.31.131.126:40413/ using document root <none> and password file <none>
I20240507 02:07:58.977330 8049 fs_manager.cc:359] Metadata directory not provided
I20240507 02:07:58.977384 8049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:07:58.977497 8049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:07:58.978861 8049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/master-0/data/instance:
uuid: "0d75f713da5242508086ac2a875bda6c"
format_stamp: "Formatted at 2024-05-07 02:07:58 on dist-test-slave-v0bg"
server_key: "7f0e1066c4b573d121b05af011b7b211"
server_key_iv: "84cc3735e271394319e856b1a1bec0d8"
server_key_version: "encryptionkey@0"
I20240507 02:07:58.979214 8049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/master-0/wal/instance:
uuid: "0d75f713da5242508086ac2a875bda6c"
format_stamp: "Formatted at 2024-05-07 02:07:58 on dist-test-slave-v0bg"
server_key: "7f0e1066c4b573d121b05af011b7b211"
server_key_iv: "84cc3735e271394319e856b1a1bec0d8"
server_key_version: "encryptionkey@0"
I20240507 02:07:58.980666 8049 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240507 02:07:58.981472 8064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:58.981665 8049 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240507 02:07:58.981770 8049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/master-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/master-0/wal
uuid: "0d75f713da5242508086ac2a875bda6c"
format_stamp: "Formatted at 2024-05-07 02:07:58 on dist-test-slave-v0bg"
server_key: "7f0e1066c4b573d121b05af011b7b211"
server_key_iv: "84cc3735e271394319e856b1a1bec0d8"
server_key_version: "encryptionkey@0"
I20240507 02:07:58.981837 8049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:07:59.019307 8049 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:07:59.021381 8049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:07:59.021490 8049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:07:59.024950 8049 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.126:39211
I20240507 02:07:59.024981 8116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.126:39211 every 8 connection(s)
I20240507 02:07:59.025274 8049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
I20240507 02:07:59.026027 8117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:07:59.028097 8117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d75f713da5242508086ac2a875bda6c: Bootstrap starting.
I20240507 02:07:59.028604 8117 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d75f713da5242508086ac2a875bda6c: Neither blocks nor log segments found. Creating new log.
I20240507 02:07:59.028841 8117 log.cc:824] T 00000000000000000000000000000000 P 0d75f713da5242508086ac2a875bda6c: Log is configured to *not* fsync() on all Append() calls
I20240507 02:07:59.029407 8117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d75f713da5242508086ac2a875bda6c: No bootstrap required, opened a new log
I20240507 02:07:59.030767 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 8049
I20240507 02:07:59.030846 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/master-0/wal/instance
I20240507 02:07:59.030851 8117 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d75f713da5242508086ac2a875bda6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d75f713da5242508086ac2a875bda6c" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 39211 } }
I20240507 02:07:59.030978 8117 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d75f713da5242508086ac2a875bda6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:07:59.030990 32269 external_mini_cluster.cc:1428] Setting key 55243a4cee9f59fb0b9a70da3b9d983b
I20240507 02:07:59.031002 8117 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d75f713da5242508086ac2a875bda6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d75f713da5242508086ac2a875bda6c, State: Initialized, Role: FOLLOWER
I20240507 02:07:59.031108 8117 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d75f713da5242508086ac2a875bda6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d75f713da5242508086ac2a875bda6c" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 39211 } }
I20240507 02:07:59.031175 8117 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d75f713da5242508086ac2a875bda6c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 02:07:59.031208 8117 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d75f713da5242508086ac2a875bda6c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 02:07:59.031247 8117 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d75f713da5242508086ac2a875bda6c [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:07:59.031803 8117 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d75f713da5242508086ac2a875bda6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d75f713da5242508086ac2a875bda6c" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 39211 } }
I20240507 02:07:59.031898 8117 leader_election.cc:302] T 00000000000000000000000000000000 P 0d75f713da5242508086ac2a875bda6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d75f713da5242508086ac2a875bda6c; no voters:
I20240507 02:07:59.032027 8117 leader_election.cc:288] T 00000000000000000000000000000000 P 0d75f713da5242508086ac2a875bda6c [CANDIDATE]: Term 1 election: Requested vote from peers
I20240507 02:07:59.032220 8122 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d75f713da5242508086ac2a875bda6c [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:07:59.032243 8117 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d75f713da5242508086ac2a875bda6c [sys.catalog]: configured and running, proceeding with master startup.
I20240507 02:07:59.032351 8122 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d75f713da5242508086ac2a875bda6c [term 1 LEADER]: Becoming Leader. State: Replica: 0d75f713da5242508086ac2a875bda6c, State: Running, Role: LEADER
I20240507 02:07:59.032433 8122 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d75f713da5242508086ac2a875bda6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d75f713da5242508086ac2a875bda6c" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 39211 } }
I20240507 02:07:59.032725 8122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d75f713da5242508086ac2a875bda6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d75f713da5242508086ac2a875bda6c. Latest consensus state: current_term: 1 leader_uuid: "0d75f713da5242508086ac2a875bda6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d75f713da5242508086ac2a875bda6c" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 39211 } } }
I20240507 02:07:59.032734 8123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d75f713da5242508086ac2a875bda6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d75f713da5242508086ac2a875bda6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d75f713da5242508086ac2a875bda6c" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 39211 } } }
I20240507 02:07:59.032816 8122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d75f713da5242508086ac2a875bda6c [sys.catalog]: This master's current role is: LEADER
I20240507 02:07:59.032827 8123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d75f713da5242508086ac2a875bda6c [sys.catalog]: This master's current role is: LEADER
I20240507 02:07:59.033560 8131 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 02:07:59.033954 8131 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240507 02:07:59.036038 8131 catalog_manager.cc:1347] Generated new cluster ID: 8e34210a000547b995d036640c163d71
I20240507 02:07:59.036092 8131 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
May 07 02:07:59 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047678, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:07:59.039541 8121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:59.031426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49538 (local address 127.31.131.126:39211)
0507 02:07:59.031670 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:59.031679 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:59.031699 (+ 20us) server_negotiation.cc:407] Connection header received
0507 02:07:59.031779 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:59.031783 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:59.031819 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:59.031946 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:59.032884 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:59.034040 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:59.034632 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:59.034874 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:59.036860 (+ 1986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:59.036886 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:59.036886 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:07:59.036900 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:59.038311 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:59.038647 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:59.038649 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:59.038650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:59.038706 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:59.038857 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:59.038857 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:59.038858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:59.039027 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:59.039145 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:59.039273 (+ 128us) server_negotiation.cc:299] Negotiation successful
0507 02:07:59.039386 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":64,"threads_started":1}
I20240507 02:07:59.057672 8131 catalog_manager.cc:1370] Generated new certificate authority record
I20240507 02:07:59.058188 8131 catalog_manager.cc:1504] Loading token signing keys...
I20240507 02:07:59.067127 8131 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 0d75f713da5242508086ac2a875bda6c: Generated new TSK 0
I20240507 02:07:59.067299 8131 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.131.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab.
Entry for principal kudu/127.31.131.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab.
I20240507 02:07:59.097249 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.65:0
--local_ip_for_outbound_sockets=127.31.131.65
--webserver_interface=127.31.131.65
--webserver_port=0
--tserver_master_addrs=127.31.131.126:39211
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab
--principal=kudu/127.31.131.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:39087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:07:59.171484 8143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:07:59.171651 8143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:07:59.171679 8143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:07:59.172906 8143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:07:59.172947 8143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:07:59.172991 8143 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.65
I20240507 02:07:59.174535 8143 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:39087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab
--principal=kudu/127.31.131.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.131.65
--webserver_port=0
--tserver_master_addrs=127.31.131.126:39211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8143
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.65
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:07:59.174764 8143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:07:59.174932 8143 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:07:59.178527 8149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:07:59.178696 8143 server_base.cc:1034] running on GCE node
W20240507 02:07:59.178527 8151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:07:59.178541 8148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:07:59.178921 8143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:07:59.179108 8143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:07:59.180236 8143 hybrid_clock.cc:648] HybridClock initialized: now 1715047679180225 us; error 30 us; skew 500 ppm
May 07 02:07:59 dist-test-slave-v0bg krb5kdc[8036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047679, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:07:59.182395 8143 init.cc:377] Logged in from keytab as kudu/127.31.131.65@KRBTEST.COM (short username kudu)
I20240507 02:07:59.182785 8143 webserver.cc:460] Webserver started at http://127.31.131.65:35769/ using document root <none> and password file <none>
I20240507 02:07:59.182945 8143 fs_manager.cc:359] Metadata directory not provided
I20240507 02:07:59.182984 8143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:07:59.183082 8143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:07:59.183970 8143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-0/data/instance:
uuid: "77b257ae3d834684bafe7e516141c7e7"
format_stamp: "Formatted at 2024-05-07 02:07:59 on dist-test-slave-v0bg"
server_key: "1e0fcb1d3f1b4319a65e5e165fc0a5d1"
server_key_iv: "0cc6ada059169df2a137961bca950614"
server_key_version: "encryptionkey@0"
I20240507 02:07:59.184257 8143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-0/wal/instance:
uuid: "77b257ae3d834684bafe7e516141c7e7"
format_stamp: "Formatted at 2024-05-07 02:07:59 on dist-test-slave-v0bg"
server_key: "1e0fcb1d3f1b4319a65e5e165fc0a5d1"
server_key_iv: "0cc6ada059169df2a137961bca950614"
server_key_version: "encryptionkey@0"
I20240507 02:07:59.185285 8143 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240507 02:07:59.185937 8158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:59.186071 8143 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240507 02:07:59.186146 8143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-0/wal
uuid: "77b257ae3d834684bafe7e516141c7e7"
format_stamp: "Formatted at 2024-05-07 02:07:59 on dist-test-slave-v0bg"
server_key: "1e0fcb1d3f1b4319a65e5e165fc0a5d1"
server_key_iv: "0cc6ada059169df2a137961bca950614"
server_key_version: "encryptionkey@0"
I20240507 02:07:59.186198 8143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:07:59.200322 8143 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:07:59.202191 8143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:07:59.202302 8143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:07:59.202466 8143 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:07:59.202745 8143 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:07:59.202777 8143 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:59.202803 8143 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:07:59.202831 8143 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:59.207841 8143 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.65:43939
I20240507 02:07:59.207892 8272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.65:43939 every 8 connection(s)
I20240507 02:07:59.208153 8143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
I20240507 02:07:59.210909 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 8143
I20240507 02:07:59.211001 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-0/wal/instance
I20240507 02:07:59.211148 32269 external_mini_cluster.cc:1428] Setting key 3425e137153169338c74743c75ea8ffb
May 07 02:07:59 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047679, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.65@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:07:59.216188 8275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:59.208759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:39211 (local address 127.31.131.65:33293)
0507 02:07:59.208905 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:59.208934 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:59.209377 (+ 443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:59.209601 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:59.209605 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:59.209868 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:07:59.210126 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:59.210130 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:59.210830 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:59.210831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:59.211238 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:59.211240 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:59.211639 (+ 399us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:59.212087 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:07:59.212102 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:07:59.213610 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:07:59.215206 (+ 1596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:59.215211 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:59.215213 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:59.215397 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:59.215545 (+ 148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:59.215546 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:59.215546 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:59.215589 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:59.215802 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:07:59.215805 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:07:59.215895 (+ 90us) client_negotiation.cc:769] Sending connection context
0507 02:07:59.215968 (+ 73us) client_negotiation.cc:240] Negotiation successful
0507 02:07:59.216059 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":76,"threads_started":1}
I20240507 02:07:59.216199 8121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:59.208807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:33293 (local address 127.31.131.126:39211)
0507 02:07:59.208867 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:59.208870 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:59.209006 (+ 136us) server_negotiation.cc:407] Connection header received
0507 02:07:59.209474 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:59.209475 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:59.209491 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:59.209544 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:59.210221 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:59.210749 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:59.211407 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:59.211603 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:59.213698 (+ 2095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:59.213703 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:59.213703 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:07:59.213710 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:59.215129 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:59.215464 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:59.215465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:59.215466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:59.215492 (+ 26us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:59.215635 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:59.215635 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:59.215636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:59.215746 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:59.215809 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:59.216007 (+ 198us) server_negotiation.cc:299] Negotiation successful
0507 02:07:59.216103 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20240507 02:07:59.216627 8273 heartbeater.cc:347] Connected to a master server at 127.31.131.126:39211
I20240507 02:07:59.216734 8273 heartbeater.cc:464] Registering TS with master...
I20240507 02:07:59.216970 8273 heartbeater.cc:511] Master 127.31.131.126:39211 requested a full tablet report, sending...
I20240507 02:07:59.217487 8081 ts_manager.cc:194] Registered new tserver with Master: 77b257ae3d834684bafe7e516141c7e7 (127.31.131.65:43939)
I20240507 02:07:59.218106 8081 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.131.65@KRBTEST.COM'} at 127.31.131.65:33293
WARNING: no policy specified for kudu/127.31.131.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab.
Entry for principal kudu/127.31.131.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab.
I20240507 02:07:59.232568 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.66:0
--local_ip_for_outbound_sockets=127.31.131.66
--webserver_interface=127.31.131.66
--webserver_port=0
--tserver_master_addrs=127.31.131.126:39211
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab
--principal=kudu/127.31.131.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:39087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:07:59.306890 8278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:07:59.307041 8278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:07:59.307061 8278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:07:59.308317 8278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:07:59.308358 8278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:07:59.308383 8278 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.66
I20240507 02:07:59.309885 8278 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:39087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab
--principal=kudu/127.31.131.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.131.66
--webserver_port=0
--tserver_master_addrs=127.31.131.126:39211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8278
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.66
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:07:59.310074 8278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:07:59.310235 8278 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:07:59.313833 8283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:07:59.313813 8284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:07:59.314086 8278 server_base.cc:1034] running on GCE node
W20240507 02:07:59.314252 8286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:07:59.314460 8278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:07:59.314668 8278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:07:59.315788 8278 hybrid_clock.cc:648] HybridClock initialized: now 1715047679315776 us; error 29 us; skew 500 ppm
May 07 02:07:59 dist-test-slave-v0bg krb5kdc[8036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047679, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:07:59.323814 8278 init.cc:377] Logged in from keytab as kudu/127.31.131.66@KRBTEST.COM (short username kudu)
I20240507 02:07:59.324180 8278 webserver.cc:460] Webserver started at http://127.31.131.66:46605/ using document root <none> and password file <none>
I20240507 02:07:59.324368 8278 fs_manager.cc:359] Metadata directory not provided
I20240507 02:07:59.324409 8278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:07:59.324504 8278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:07:59.325417 8278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-1/data/instance:
uuid: "e63bc5a930394106b415eac652d414b4"
format_stamp: "Formatted at 2024-05-07 02:07:59 on dist-test-slave-v0bg"
server_key: "aad2fcaaded9a33ab6988bb7d0c24125"
server_key_iv: "8d99acb3ec7711057e6b15442d8fb3cf"
server_key_version: "encryptionkey@0"
I20240507 02:07:59.325714 8278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-1/wal/instance:
uuid: "e63bc5a930394106b415eac652d414b4"
format_stamp: "Formatted at 2024-05-07 02:07:59 on dist-test-slave-v0bg"
server_key: "aad2fcaaded9a33ab6988bb7d0c24125"
server_key_iv: "8d99acb3ec7711057e6b15442d8fb3cf"
server_key_version: "encryptionkey@0"
I20240507 02:07:59.326810 8278 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240507 02:07:59.327409 8293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:59.327548 8278 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:59.327616 8278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-1/wal
uuid: "e63bc5a930394106b415eac652d414b4"
format_stamp: "Formatted at 2024-05-07 02:07:59 on dist-test-slave-v0bg"
server_key: "aad2fcaaded9a33ab6988bb7d0c24125"
server_key_iv: "8d99acb3ec7711057e6b15442d8fb3cf"
server_key_version: "encryptionkey@0"
I20240507 02:07:59.327670 8278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:07:59.354238 8278 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:07:59.356226 8278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:07:59.356341 8278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:07:59.356525 8278 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:07:59.356801 8278 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:07:59.356832 8278 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:59.356860 8278 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:07:59.356881 8278 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:59.361621 8278 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.66:34777
I20240507 02:07:59.361685 8407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.66:34777 every 8 connection(s)
I20240507 02:07:59.361925 8278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
I20240507 02:07:59.366218 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 8278
I20240507 02:07:59.366318 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-1/wal/instance
May 07 02:07:59 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047679, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.66@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:07:59.366475 32269 external_mini_cluster.cc:1428] Setting key 80f8d680f4f389109cb2a19dfae86b0f
I20240507 02:07:59.369294 8121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:59.362543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.66:48035 (local address 127.31.131.126:39211)
0507 02:07:59.362629 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:59.362632 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:59.362759 (+ 127us) server_negotiation.cc:407] Connection header received
0507 02:07:59.363193 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:59.363194 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:59.363203 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:59.363251 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:59.363896 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:59.364375 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:59.364853 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:59.364952 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:59.366944 (+ 1992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:59.366948 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:59.366948 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:07:59.366955 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:59.368157 (+ 1202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:59.368557 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:59.368558 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:59.368559 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:59.368586 (+ 27us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:59.368737 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:59.368737 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:59.368738 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:59.368846 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:59.368908 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:59.369126 (+ 218us) server_negotiation.cc:299] Negotiation successful
0507 02:07:59.369213 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20240507 02:07:59.369342 8410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:59.362520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:39211 (local address 127.31.131.66:48035)
0507 02:07:59.362661 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:59.362684 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:59.363113 (+ 429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:59.363295 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:59.363298 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:59.363557 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:07:59.363834 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:59.363838 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:59.364436 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:59.364436 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:59.364799 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:59.364800 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:59.365082 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:59.365470 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:07:59.365483 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:07:59.366857 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:07:59.368231 (+ 1374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:59.368237 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:59.368239 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:59.368490 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:59.368644 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:59.368645 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:59.368645 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:59.368691 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:59.368897 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:07:59.368900 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:07:59.368993 (+ 93us) client_negotiation.cc:769] Sending connection context
0507 02:07:59.369099 (+ 106us) client_negotiation.cc:240] Negotiation successful
0507 02:07:59.369208 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":86,"threads_started":1}
I20240507 02:07:59.369730 8408 heartbeater.cc:347] Connected to a master server at 127.31.131.126:39211
I20240507 02:07:59.369819 8408 heartbeater.cc:464] Registering TS with master...
I20240507 02:07:59.369989 8408 heartbeater.cc:511] Master 127.31.131.126:39211 requested a full tablet report, sending...
I20240507 02:07:59.370344 8081 ts_manager.cc:194] Registered new tserver with Master: e63bc5a930394106b415eac652d414b4 (127.31.131.66:34777)
I20240507 02:07:59.370875 8081 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.131.66@KRBTEST.COM'} at 127.31.131.66:48035
WARNING: no policy specified for kudu/127.31.131.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab.
Entry for principal kudu/127.31.131.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab.
I20240507 02:07:59.387102 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.67:0
--local_ip_for_outbound_sockets=127.31.131.67
--webserver_interface=127.31.131.67
--webserver_port=0
--tserver_master_addrs=127.31.131.126:39211
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab
--principal=kudu/127.31.131.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:39087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:07:59.461550 8413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:07:59.461704 8413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:07:59.461725 8413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:07:59.462985 8413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:07:59.463022 8413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:07:59.463045 8413 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.67
I20240507 02:07:59.464543 8413 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:39087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab
--principal=kudu/127.31.131.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.131.67
--webserver_port=0
--tserver_master_addrs=127.31.131.126:39211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.67
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:07:59.464702 8413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:07:59.464843 8413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:07:59.468416 8419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:07:59.468420 8418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:07:59.468446 8413 server_base.cc:1034] running on GCE node
W20240507 02:07:59.468569 8421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:07:59.468844 8413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:07:59.469053 8413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:07:59.470199 8413 hybrid_clock.cc:648] HybridClock initialized: now 1715047679470178 us; error 24 us; skew 500 ppm
May 07 02:07:59 dist-test-slave-v0bg krb5kdc[8036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047679, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:07:59.472414 8413 init.cc:377] Logged in from keytab as kudu/127.31.131.67@KRBTEST.COM (short username kudu)
I20240507 02:07:59.472759 8413 webserver.cc:460] Webserver started at http://127.31.131.67:38813/ using document root <none> and password file <none>
I20240507 02:07:59.472903 8413 fs_manager.cc:359] Metadata directory not provided
I20240507 02:07:59.472936 8413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:07:59.473009 8413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:07:59.473887 8413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-2/data/instance:
uuid: "2cf365d4ab934b3890491bc715ab5691"
format_stamp: "Formatted at 2024-05-07 02:07:59 on dist-test-slave-v0bg"
server_key: "8cc8ed6dee0e2c877460c05ed73c915c"
server_key_iv: "3013a95d2875d465ce84c3008ec4c260"
server_key_version: "encryptionkey@0"
I20240507 02:07:59.474165 8413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-2/wal/instance:
uuid: "2cf365d4ab934b3890491bc715ab5691"
format_stamp: "Formatted at 2024-05-07 02:07:59 on dist-test-slave-v0bg"
server_key: "8cc8ed6dee0e2c877460c05ed73c915c"
server_key_iv: "3013a95d2875d465ce84c3008ec4c260"
server_key_version: "encryptionkey@0"
I20240507 02:07:59.475229 8413 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240507 02:07:59.475796 8428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:59.475958 8413 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240507 02:07:59.476023 8413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-2/wal
uuid: "2cf365d4ab934b3890491bc715ab5691"
format_stamp: "Formatted at 2024-05-07 02:07:59 on dist-test-slave-v0bg"
server_key: "8cc8ed6dee0e2c877460c05ed73c915c"
server_key_iv: "3013a95d2875d465ce84c3008ec4c260"
server_key_version: "encryptionkey@0"
I20240507 02:07:59.476069 8413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:07:59.483266 8413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:07:59.485013 8413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:07:59.485096 8413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:07:59.485227 8413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:07:59.485474 8413 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:07:59.485504 8413 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:59.485524 8413 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:07:59.485546 8413 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:59.490350 8413 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.67:44273
I20240507 02:07:59.490396 8542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.67:44273 every 8 connection(s)
I20240507 02:07:59.490664 8413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
May 07 02:07:59 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047679, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.67@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:07:59.497782 8121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:59.491274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.67:45309 (local address 127.31.131.126:39211)
0507 02:07:59.491333 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:59.491335 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:59.491471 (+ 136us) server_negotiation.cc:407] Connection header received
0507 02:07:59.491918 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:59.491919 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:59.491931 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:59.491980 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:59.492651 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:59.493128 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:59.493640 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:59.493807 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:59.495702 (+ 1895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:07:59.495707 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:07:59.495707 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:07:59.495713 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:07:59.496741 (+ 1028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:59.497062 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:59.497064 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:59.497064 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:59.497086 (+ 22us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:07:59.497233 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:07:59.497234 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:07:59.497234 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:07:59.497329 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:07:59.497383 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0507 02:07:59.497601 (+ 218us) server_negotiation.cc:299] Negotiation successful
0507 02:07:59.497688 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20240507 02:07:59.497870 8545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:59.491238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:39211 (local address 127.31.131.67:45309)
0507 02:07:59.491377 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:59.491400 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:59.491824 (+ 424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:59.492033 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:59.492036 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:59.492304 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:07:59.492564 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:59.492568 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:59.493203 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:59.493204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:59.493568 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:59.493570 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:59.493914 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:59.494321 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:07:59.494333 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:07:59.495611 (+ 1278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:07:59.496811 (+ 1200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:59.496816 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:59.496819 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:59.496998 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:59.497136 (+ 138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:07:59.497136 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:07:59.497137 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:07:59.497185 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:07:59.497379 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:07:59.497384 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:07:59.497489 (+ 105us) client_negotiation.cc:769] Sending connection context
0507 02:07:59.497575 (+ 86us) client_negotiation.cc:240] Negotiation successful
0507 02:07:59.497693 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":77,"threads_started":1}
I20240507 02:07:59.498282 8543 heartbeater.cc:347] Connected to a master server at 127.31.131.126:39211
I20240507 02:07:59.498369 8543 heartbeater.cc:464] Registering TS with master...
I20240507 02:07:59.498538 8543 heartbeater.cc:511] Master 127.31.131.126:39211 requested a full tablet report, sending...
I20240507 02:07:59.498950 8081 ts_manager.cc:194] Registered new tserver with Master: 2cf365d4ab934b3890491bc715ab5691 (127.31.131.67:44273)
I20240507 02:07:59.499284 8081 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.131.67@KRBTEST.COM'} at 127.31.131.67:45309
I20240507 02:07:59.500584 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 8413
I20240507 02:07:59.500667 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-2/wal/instance
I20240507 02:07:59.500806 32269 external_mini_cluster.cc:1428] Setting key a6e2c747c42406ad5e4aea74fd16bb76
I20240507 02:07:59.501518 32269 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.31.131.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.131.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.131.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/krb5kdc/oryx_127.31.131.65.keytab.
Entry for principal oryx/127.31.131.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/krb5kdc/oryx_127.31.131.65.keytab.
WARNING: no policy specified for oryx/127.31.131.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.131.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.131.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/krb5kdc/oryx_127.31.131.66.keytab.
Entry for principal oryx/127.31.131.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/krb5kdc/oryx_127.31.131.66.keytab.
WARNING: no policy specified for oryx/127.31.131.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.131.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.131.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/krb5kdc/oryx_127.31.131.67.keytab.
Entry for principal oryx/127.31.131.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/krb5kdc/oryx_127.31.131.67.keytab.
I20240507 02:07:59.556511 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 8143
I20240507 02:07:59.560451 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 8278
I20240507 02:07:59.563951 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 8413
I20240507 02:07:59.567337 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 8049
I20240507 02:07:59.571543 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:39211
--webserver_interface=127.31.131.126
--webserver_port=40413
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab
--principal=kudu/127.31.131.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:39087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.131.126:39211
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:07:59.645795 8552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:07:59.645962 8552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:07:59.645982 8552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:07:59.647205 8552 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240507 02:07:59.647239 8552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:07:59.647250 8552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:07:59.647261 8552 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240507 02:07:59.647271 8552 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240507 02:07:59.648716 8552 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:39087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.131.126:39211
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab
--principal=kudu/127.31.131.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:39211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.131.126
--webserver_port=40413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:07:59.648871 8552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:07:59.648999 8552 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:07:59.652395 8558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:07:59.652490 8560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:07:59.652592 8552 server_base.cc:1034] running on GCE node
W20240507 02:07:59.652804 8557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:07:59.653007 8552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:07:59.653229 8552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:07:59.654362 8552 hybrid_clock.cc:648] HybridClock initialized: now 1715047679654345 us; error 27 us; skew 500 ppm
May 07 02:07:59 dist-test-slave-v0bg krb5kdc[8036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047679, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:07:59.656539 8552 init.cc:377] Logged in from keytab as kudu/127.31.131.126@KRBTEST.COM (short username kudu)
I20240507 02:07:59.656899 8552 webserver.cc:460] Webserver started at http://127.31.131.126:40413/ using document root <none> and password file <none>
I20240507 02:07:59.657058 8552 fs_manager.cc:359] Metadata directory not provided
I20240507 02:07:59.657116 8552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:07:59.658260 8552 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20240507 02:07:59.658828 8567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:59.658967 8552 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:59.659039 8552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/master-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/master-0/wal
uuid: "0d75f713da5242508086ac2a875bda6c"
format_stamp: "Formatted at 2024-05-07 02:07:58 on dist-test-slave-v0bg"
server_key: "7f0e1066c4b573d121b05af011b7b211"
server_key_iv: "84cc3735e271394319e856b1a1bec0d8"
server_key_version: "encryptionkey@0"
I20240507 02:07:59.659296 8552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:07:59.674276 8552 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:07:59.675788 8552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:07:59.675902 8552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:07:59.679272 8552 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.126:39211
I20240507 02:07:59.679308 8620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.126:39211 every 8 connection(s)
I20240507 02:07:59.679612 8552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
I20240507 02:07:59.680871 8621 sys_catalog.cc:263] Verifying existing consensus state
I20240507 02:07:59.681376 8621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d75f713da5242508086ac2a875bda6c: Bootstrap starting.
I20240507 02:07:59.684846 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 8552
I20240507 02:07:59.685092 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.65:43939
--local_ip_for_outbound_sockets=127.31.131.65
--tserver_master_addrs=127.31.131.126:39211
--webserver_port=35769
--webserver_interface=127.31.131.65
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab
--principal=kudu/127.31.131.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:39087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.31.131.65
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/krb5kdc/oryx_127.31.131.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240507 02:07:59.689975 8621 log.cc:824] T 00000000000000000000000000000000 P 0d75f713da5242508086ac2a875bda6c: Log is configured to *not* fsync() on all Append() calls
I20240507 02:07:59.691437 8621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d75f713da5242508086ac2a875bda6c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240507 02:07:59.691661 8621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d75f713da5242508086ac2a875bda6c: Bootstrap complete.
I20240507 02:07:59.693511 8621 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d75f713da5242508086ac2a875bda6c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d75f713da5242508086ac2a875bda6c" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 39211 } }
I20240507 02:07:59.693692 8621 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d75f713da5242508086ac2a875bda6c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d75f713da5242508086ac2a875bda6c, State: Initialized, Role: FOLLOWER
I20240507 02:07:59.693796 8621 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d75f713da5242508086ac2a875bda6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d75f713da5242508086ac2a875bda6c" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 39211 } }
I20240507 02:07:59.693871 8621 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d75f713da5242508086ac2a875bda6c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 02:07:59.693907 8621 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d75f713da5242508086ac2a875bda6c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 02:07:59.693964 8621 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d75f713da5242508086ac2a875bda6c [term 1 FOLLOWER]: Advancing to term 2
I20240507 02:07:59.694808 8621 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d75f713da5242508086ac2a875bda6c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d75f713da5242508086ac2a875bda6c" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 39211 } }
I20240507 02:07:59.694892 8621 leader_election.cc:302] T 00000000000000000000000000000000 P 0d75f713da5242508086ac2a875bda6c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d75f713da5242508086ac2a875bda6c; no voters:
I20240507 02:07:59.695027 8621 leader_election.cc:288] T 00000000000000000000000000000000 P 0d75f713da5242508086ac2a875bda6c [CANDIDATE]: Term 2 election: Requested vote from peers
I20240507 02:07:59.695080 8625 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d75f713da5242508086ac2a875bda6c [term 2 FOLLOWER]: Leader election won for term 2
I20240507 02:07:59.695191 8625 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d75f713da5242508086ac2a875bda6c [term 2 LEADER]: Becoming Leader. State: Replica: 0d75f713da5242508086ac2a875bda6c, State: Running, Role: LEADER
I20240507 02:07:59.695240 8621 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d75f713da5242508086ac2a875bda6c [sys.catalog]: configured and running, proceeding with master startup.
I20240507 02:07:59.695261 8625 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d75f713da5242508086ac2a875bda6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d75f713da5242508086ac2a875bda6c" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 39211 } }
I20240507 02:07:59.695569 8627 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d75f713da5242508086ac2a875bda6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0d75f713da5242508086ac2a875bda6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d75f713da5242508086ac2a875bda6c" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 39211 } } }
I20240507 02:07:59.695582 8625 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d75f713da5242508086ac2a875bda6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d75f713da5242508086ac2a875bda6c. Latest consensus state: current_term: 2 leader_uuid: "0d75f713da5242508086ac2a875bda6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d75f713da5242508086ac2a875bda6c" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 39211 } } }
I20240507 02:07:59.695838 8627 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d75f713da5242508086ac2a875bda6c [sys.catalog]: This master's current role is: LEADER
I20240507 02:07:59.695876 8625 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d75f713da5242508086ac2a875bda6c [sys.catalog]: This master's current role is: LEADER
I20240507 02:07:59.696167 8634 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 02:07:59.696653 8634 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240507 02:07:59.696810 8634 catalog_manager.cc:1259] Loaded cluster ID: 8e34210a000547b995d036640c163d71
I20240507 02:07:59.696933 8634 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240507 02:07:59.698354 8634 catalog_manager.cc:1504] Loading token signing keys...
I20240507 02:07:59.698575 8634 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d75f713da5242508086ac2a875bda6c: Loaded TSK: 0
I20240507 02:07:59.698803 8634 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20240507 02:07:59.762077 8623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:07:59.762255 8623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:07:59.762285 8623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:07:59.763626 8623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:07:59.763674 8623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:07:59.763698 8623 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.65
I20240507 02:07:59.765237 8623 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:39087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/krb5kdc/oryx_127.31.131.65.keytab
--principal=oryx/127.31.131.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.65:43939
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.131.65
--webserver_port=35769
--tserver_master_addrs=127.31.131.126:39211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.65
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:07:59.765429 8623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:07:59.765612 8623 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:07:59.769294 8648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:07:59.769349 8650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:07:59.769352 8647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:07:59.769685 8623 server_base.cc:1034] running on GCE node
I20240507 02:07:59.769843 8623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:07:59.770042 8623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:07:59.771178 8623 hybrid_clock.cc:648] HybridClock initialized: now 1715047679771164 us; error 33 us; skew 500 ppm
May 07 02:07:59 dist-test-slave-v0bg krb5kdc[8036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047679, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.131.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:07:59.773595 8623 init.cc:377] Logged in from keytab as oryx/127.31.131.65@KRBTEST.COM (short username oryx)
I20240507 02:07:59.774029 8623 webserver.cc:460] Webserver started at http://127.31.131.65:35769/ using document root <none> and password file <none>
I20240507 02:07:59.774202 8623 fs_manager.cc:359] Metadata directory not provided
I20240507 02:07:59.774258 8623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:07:59.775496 8623 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20240507 02:07:59.775992 8657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:59.776151 8623 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:59.776234 8623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-0/wal
uuid: "77b257ae3d834684bafe7e516141c7e7"
format_stamp: "Formatted at 2024-05-07 02:07:59 on dist-test-slave-v0bg"
server_key: "1e0fcb1d3f1b4319a65e5e165fc0a5d1"
server_key_iv: "0cc6ada059169df2a137961bca950614"
server_key_version: "encryptionkey@0"
I20240507 02:07:59.776504 8623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:07:59.800004 8623 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:07:59.801899 8623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:07:59.802001 8623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:07:59.802225 8623 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:07:59.802539 8623 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:07:59.802605 8623 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:59.802630 8623 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:07:59.802659 8623 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:59.809334 8623 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.65:43939
I20240507 02:07:59.809445 8770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.65:43939 every 8 connection(s)
I20240507 02:07:59.809696 8623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
May 07 02:07:59 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.65@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
May 07 02:07:59 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.65@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
I20240507 02:07:59.817111 8773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:59.810401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:39211 (local address 127.31.131.65:50223)
0507 02:07:59.810573 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:59.810601 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:59.811030 (+ 429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:59.811445 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:59.811449 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:59.811752 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:07:59.812019 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:59.812022 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:59.813520 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:59.813522 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:59.813912 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:59.813914 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:59.814263 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:59.814818 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:07:59.814832 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:07:59.816975 (+ 2143us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.131.126:39211: Server oryx/127.31.131.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":72,"threads_started":1}
W20240507 02:07:59.817296 8771 heartbeater.cc:644] Failed to heartbeat to 127.31.131.126:39211 (0 consecutive failures): Not authorized: Failed to ping master at 127.31.131.126:39211: Client connection negotiation failed: client connection to 127.31.131.126:39211: Server oryx/127.31.131.126@KRBTEST.COM not found in Kerberos database
W20240507 02:07:59.817675 8774 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:07:59.810482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:50223 (local address 127.31.131.126:39211)
0507 02:07:59.810668 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:59.810678 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:59.810703 (+ 25us) server_negotiation.cc:407] Connection header received
0507 02:07:59.811213 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:59.811219 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:59.811266 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:59.811427 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:59.812137 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:59.813405 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:59.814013 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:59.814232 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:59.817145 (+ 2913us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.131.65:50223: BlockingRecv error: recv got EOF from 127.31.131.65:50223 (error 108)
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":103,"threads_started":1}
I20240507 02:07:59.819429 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 8623
I20240507 02:07:59.820060 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.66:34777
--local_ip_for_outbound_sockets=127.31.131.66
--tserver_master_addrs=127.31.131.126:39211
--webserver_port=46605
--webserver_interface=127.31.131.66
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab
--principal=kudu/127.31.131.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:39087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.31.131.66
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/krb5kdc/oryx_127.31.131.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
May 07 02:07:59 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.65@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
May 07 02:07:59 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.65@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
I20240507 02:07:59.822841 8773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:59.817532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:39211 (local address 127.31.131.65:56363)
0507 02:07:59.817582 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:59.817593 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:59.817641 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:59.817920 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:59.817921 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:59.818046 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:07:59.818185 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:59.818186 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:59.818909 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:59.818910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:59.819262 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:59.819264 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:59.819330 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:59.819784 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:07:59.819787 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:07:59.822779 (+ 2992us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.131.126:39211: Server oryx/127.31.131.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":32}
W20240507 02:07:59.822948 8775 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:07:59.817598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:56363 (local address 127.31.131.126:39211)
0507 02:07:59.817806 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:59.817807 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:59.817819 (+ 12us) server_negotiation.cc:407] Connection header received
0507 02:07:59.817840 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:59.817841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:59.817852 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:59.817915 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:59.818279 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:59.818839 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:59.819333 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:59.819509 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:59.822880 (+ 3371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.131.65:56363: BlockingRecv error: recv got EOF from 127.31.131.65:56363 (error 108)
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":85,"threads_started":1}
May 07 02:07:59 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.65@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
May 07 02:07:59 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.65@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
I20240507 02:07:59.827801 8773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:59.823376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:39211 (local address 127.31.131.65:37049)
0507 02:07:59.823471 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:59.823481 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:59.823525 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:59.823665 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:59.823666 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:59.823799 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:07:59.823959 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:59.823961 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:59.824790 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:59.824791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:59.825319 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:59.825321 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:59.825386 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:59.825913 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:07:59.825918 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:07:59.827742 (+ 1824us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.131.126:39211: Server oryx/127.31.131.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":81}
W20240507 02:07:59.827913 8775 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:07:59.823432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:37049 (local address 127.31.131.126:39211)
0507 02:07:59.823471 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:59.823473 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:59.823515 (+ 42us) server_negotiation.cc:407] Connection header received
0507 02:07:59.823574 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:59.823575 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:59.823587 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:59.823662 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:59.824038 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:59.824700 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:59.825400 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:59.825571 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:59.827856 (+ 2285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.131.65:37049: BlockingRecv error: recv got EOF from 127.31.131.65:37049 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20240507 02:07:59.828177 8771 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20240507 02:07:59.898133 8776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:07:59.898310 8776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:07:59.898331 8776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:07:59.899609 8776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:07:59.899662 8776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:07:59.899690 8776 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.66
I20240507 02:07:59.901544 8776 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:39087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/krb5kdc/oryx_127.31.131.66.keytab
--principal=oryx/127.31.131.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.66:34777
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.131.66
--webserver_port=46605
--tserver_master_addrs=127.31.131.126:39211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.66
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:07:59.901751 8776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:07:59.901929 8776 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240507 02:07:59.905608 8776 server_base.cc:1034] running on GCE node
W20240507 02:07:59.905594 8784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:07:59.905743 8782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:07:59.905596 8781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:07:59.905949 8776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:07:59.906116 8776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:07:59.907250 8776 hybrid_clock.cc:648] HybridClock initialized: now 1715047679907236 us; error 29 us; skew 500 ppm
May 07 02:07:59 dist-test-slave-v0bg krb5kdc[8036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047679, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.131.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:07:59.909513 8776 init.cc:377] Logged in from keytab as oryx/127.31.131.66@KRBTEST.COM (short username oryx)
I20240507 02:07:59.909893 8776 webserver.cc:460] Webserver started at http://127.31.131.66:46605/ using document root <none> and password file <none>
I20240507 02:07:59.910044 8776 fs_manager.cc:359] Metadata directory not provided
I20240507 02:07:59.910092 8776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:07:59.911657 8776 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20240507 02:07:59.912338 8791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:59.912547 8776 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240507 02:07:59.912640 8776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-1/wal
uuid: "e63bc5a930394106b415eac652d414b4"
format_stamp: "Formatted at 2024-05-07 02:07:59 on dist-test-slave-v0bg"
server_key: "aad2fcaaded9a33ab6988bb7d0c24125"
server_key_iv: "8d99acb3ec7711057e6b15442d8fb3cf"
server_key_version: "encryptionkey@0"
I20240507 02:07:59.912972 8776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:07:59.932147 8776 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:07:59.934070 8776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:07:59.934170 8776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:07:59.934367 8776 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:07:59.934634 8776 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:07:59.934664 8776 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:59.934690 8776 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:07:59.934705 8776 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:07:59.939498 8776 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.66:34777
I20240507 02:07:59.939599 8904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.66:34777 every 8 connection(s)
I20240507 02:07:59.939847 8776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
May 07 02:07:59 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.66@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
I20240507 02:07:59.944486 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 8776
I20240507 02:07:59.944747 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.67:44273
--local_ip_for_outbound_sockets=127.31.131.67
--tserver_master_addrs=127.31.131.126:39211
--webserver_port=38813
--webserver_interface=127.31.131.67
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab
--principal=kudu/127.31.131.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:39087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.31.131.67
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/krb5kdc/oryx_127.31.131.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
May 07 02:07:59 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.66@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
I20240507 02:07:59.945585 8907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:59.940536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:39211 (local address 127.31.131.66:56489)
0507 02:07:59.940679 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:59.940704 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:59.941139 (+ 435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:59.941336 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:59.941339 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:59.941606 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:07:59.941892 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:59.941896 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:59.942480 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:59.942480 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:59.942873 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:59.942874 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:59.943152 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:59.943554 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:07:59.943565 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:07:59.945473 (+ 1908us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.131.126:39211: Server oryx/127.31.131.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":77,"threads_started":1}
W20240507 02:07:59.945648 8775 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:07:59.940554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.66:56489 (local address 127.31.131.126:39211)
0507 02:07:59.940603 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:59.940605 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:59.940787 (+ 182us) server_negotiation.cc:407] Connection header received
0507 02:07:59.941220 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:59.941221 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:59.941231 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:59.941280 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:59.941959 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:59.942422 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:59.942929 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:59.943034 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:59.945591 (+ 2557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.131.66:56489: BlockingRecv error: recv got EOF from 127.31.131.66:56489 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20240507 02:07:59.945839 8905 heartbeater.cc:644] Failed to heartbeat to 127.31.131.126:39211 (0 consecutive failures): Not authorized: Failed to ping master at 127.31.131.126:39211: Client connection negotiation failed: client connection to 127.31.131.126:39211: Server oryx/127.31.131.126@KRBTEST.COM not found in Kerberos database
May 07 02:07:59 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.66@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
May 07 02:07:59 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.66@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
I20240507 02:07:59.949478 8907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:59.946033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:39211 (local address 127.31.131.66:35925)
0507 02:07:59.946080 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:59.946088 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:59.946120 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:59.946240 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:59.946241 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:59.946346 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:07:59.946478 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:59.946480 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:59.947075 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:59.947076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:59.947459 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:59.947461 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:59.947520 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:59.947965 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:07:59.947968 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:07:59.949437 (+ 1469us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.131.126:39211: Server oryx/127.31.131.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20240507 02:07:59.949602 8775 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:07:59.946095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.66:35925 (local address 127.31.131.126:39211)
0507 02:07:59.946148 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:59.946150 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:59.946158 (+ 8us) server_negotiation.cc:407] Connection header received
0507 02:07:59.946178 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:59.946179 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:59.946189 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:59.946231 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:59.946564 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:59.947010 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:59.947532 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:59.947681 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:59.949543 (+ 1862us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.131.66:35925: BlockingRecv error: recv got EOF from 127.31.131.66:35925 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
May 07 02:07:59 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.66@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
May 07 02:07:59 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.66@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
I20240507 02:07:59.953068 8907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:07:59.949677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:39211 (local address 127.31.131.66:53195)
0507 02:07:59.949783 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0507 02:07:59.949791 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0507 02:07:59.949830 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:07:59.949929 (+ 99us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:07:59.949930 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:07:59.950031 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:07:59.950135 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:59.950136 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:59.950735 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:07:59.950736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:07:59.951051 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:07:59.951052 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:07:59.951103 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:59.951512 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:07:59.951516 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:07:59.953009 (+ 1493us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.131.126:39211: Server oryx/127.31.131.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":71,"threads_started":1}
W20240507 02:07:59.953140 8775 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:07:59.949729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.66:53195 (local address 127.31.131.126:39211)
0507 02:07:59.949760 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0507 02:07:59.949762 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:07:59.949838 (+ 76us) server_negotiation.cc:407] Connection header received
0507 02:07:59.949865 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:07:59.949866 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:07:59.949875 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:07:59.949921 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:07:59.950194 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:59.950662 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:07:59.951109 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:07:59.951263 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:07:59.953093 (+ 1830us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.131.66:53195: BlockingRecv error: recv got EOF from 127.31.131.66:53195 (error 108)
Metrics: {"server-negotiator.queue_time_us":12}
W20240507 02:07:59.953272 8905 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20240507 02:08:00.020799 8908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:08:00.020972 8908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:08:00.021003 8908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:08:00.022279 8908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:08:00.022327 8908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:08:00.022361 8908 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.67
I20240507 02:08:00.023916 8908 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:39087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/krb5kdc/oryx_127.31.131.67.keytab
--principal=oryx/127.31.131.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.67:44273
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.131.67
--webserver_port=38813
--tserver_master_addrs=127.31.131.126:39211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8908
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.67
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:08:00.024116 8908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:08:00.024292 8908 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:08:00.027935 8914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:08:00.028092 8908 server_base.cc:1034] running on GCE node
W20240507 02:08:00.028136 8915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:08:00.027963 8917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:08:00.028348 8908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:08:00.028522 8908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:08:00.029640 8908 hybrid_clock.cc:648] HybridClock initialized: now 1715047680029628 us; error 28 us; skew 500 ppm
May 07 02:08:00 dist-test-slave-v0bg krb5kdc[8036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047680, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.131.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:08:00.031800 8908 init.cc:377] Logged in from keytab as oryx/127.31.131.67@KRBTEST.COM (short username oryx)
I20240507 02:08:00.032172 8908 webserver.cc:460] Webserver started at http://127.31.131.67:38813/ using document root <none> and password file <none>
I20240507 02:08:00.032334 8908 fs_manager.cc:359] Metadata directory not provided
I20240507 02:08:00.032387 8908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:08:00.033532 8908 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20240507 02:08:00.033991 8924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:08:00.034159 8908 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240507 02:08:00.034225 8908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-2/wal
uuid: "2cf365d4ab934b3890491bc715ab5691"
format_stamp: "Formatted at 2024-05-07 02:07:59 on dist-test-slave-v0bg"
server_key: "8cc8ed6dee0e2c877460c05ed73c915c"
server_key_iv: "3013a95d2875d465ce84c3008ec4c260"
server_key_version: "encryptionkey@0"
I20240507 02:08:00.034492 8908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:08:00.051666 8908 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:08:00.053593 8908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:08:00.053697 8908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:08:00.053880 8908 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:08:00.054167 8908 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:08:00.054198 8908 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:08:00.054225 8908 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:08:00.054241 8908 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:08:00.059018 8908 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.67:44273
I20240507 02:08:00.059079 9037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.67:44273 every 8 connection(s)
I20240507 02:08:00.059355 8908 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
May 07 02:08:00 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.67@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
May 07 02:08:00 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.67@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
I20240507 02:08:00.064947 9040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:00.059915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:39211 (local address 127.31.131.67:58995)
0507 02:08:00.060056 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0507 02:08:00.060083 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0507 02:08:00.060516 (+ 433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:08:00.060719 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:08:00.060723 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:08:00.060996 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:08:00.061285 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:00.061289 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:00.061912 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:08:00.061913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:08:00.062287 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:00.062288 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:00.062609 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:00.063046 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:08:00.063059 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:08:00.064833 (+ 1774us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.131.126:39211: Server oryx/127.31.131.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":72,"threads_started":1}
W20240507 02:08:00.064965 8775 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:08:00.059947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.67:58995 (local address 127.31.131.126:39211)
0507 02:08:00.059999 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:00.060001 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:00.060155 (+ 154us) server_negotiation.cc:407] Connection header received
0507 02:08:00.060596 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:00.060597 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:00.060606 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:00.060664 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:08:00.061352 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:00.061849 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:08:00.062350 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:00.062464 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:00.064920 (+ 2456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.131.67:58995: BlockingRecv error: recv got EOF from 127.31.131.67:58995 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20240507 02:08:00.065140 9038 heartbeater.cc:644] Failed to heartbeat to 127.31.131.126:39211 (0 consecutive failures): Not authorized: Failed to ping master at 127.31.131.126:39211: Client connection negotiation failed: client connection to 127.31.131.126:39211: Server oryx/127.31.131.126@KRBTEST.COM not found in Kerberos database
May 07 02:08:00 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.67@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
May 07 02:08:00 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.67@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
I20240507 02:08:00.068476 9040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:00.065336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:39211 (local address 127.31.131.67:51579)
0507 02:08:00.065380 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0507 02:08:00.065390 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0507 02:08:00.065420 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:08:00.065611 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:08:00.065612 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:08:00.065711 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:08:00.065847 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:00.065850 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:00.066381 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:08:00.066381 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:08:00.066739 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:00.066740 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:00.066788 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:00.067136 (+ 348us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:08:00.067139 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:08:00.068435 (+ 1296us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.131.126:39211: Server oryx/127.31.131.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20240507 02:08:00.068562 8775 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:08:00.065475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.67:51579 (local address 127.31.131.126:39211)
0507 02:08:00.065521 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:00.065523 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:00.065529 (+ 6us) server_negotiation.cc:407] Connection header received
0507 02:08:00.065547 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:00.065548 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:00.065555 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:00.065599 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:08:00.065901 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:00.066333 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:08:00.066790 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:00.066894 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:00.068512 (+ 1618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.131.67:51579: BlockingRecv error: recv got EOF from 127.31.131.67:51579 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
I20240507 02:08:00.068746 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 8908
May 07 02:08:00 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.67@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
May 07 02:08:00 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.67@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
I20240507 02:08:00.072826 9040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:00.068933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:39211 (local address 127.31.131.67:41195)
0507 02:08:00.068977 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0507 02:08:00.068987 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0507 02:08:00.069021 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:08:00.069171 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:08:00.069171 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:08:00.069282 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:08:00.069413 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:00.069414 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:00.070222 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:08:00.070223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:08:00.070743 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:00.070744 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:00.070815 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:00.071206 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:08:00.071210 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:08:00.072760 (+ 1550us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.131.126:39211: Server oryx/127.31.131.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20240507 02:08:00.072876 8775 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:08:00.068975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.67:41195 (local address 127.31.131.126:39211)
0507 02:08:00.069007 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:00.069009 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:00.069017 (+ 8us) server_negotiation.cc:407] Connection header received
0507 02:08:00.069102 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:00.069103 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:00.069110 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:00.069154 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:08:00.069478 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:00.070153 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:08:00.070811 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:00.070917 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:00.072832 (+ 1915us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.131.67:41195: BlockingRecv error: recv got EOF from 127.31.131.67:41195 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20240507 02:08:00.072978 9038 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20240507 02:08:00.075220 8774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:00.069051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49542 (local address 127.31.131.126:39211)
0507 02:08:00.069077 (+ 26us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:00.069078 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:00.069356 (+ 278us) server_negotiation.cc:407] Connection header received
0507 02:08:00.069381 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:00.069381 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:00.069393 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:00.069456 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:08:00.070248 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:00.070949 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:08:00.071567 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:00.071677 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:00.072671 (+ 994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:08:00.072678 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:08:00.072678 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:08:00.072710 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:08:00.074282 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:08:00.074586 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:08:00.074589 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:08:00.074590 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:08:00.074637 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:08:00.074739 (+ 102us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:08:00.074739 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:08:00.074739 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:08:00.074890 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:08:00.074978 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0507 02:08:00.075056 (+ 78us) server_negotiation.cc:299] Negotiation successful
0507 02:08:00.075142 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
May 07 02:08:00 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.65@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
May 07 02:08:00 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.65@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
I20240507 02:08:00.832499 9042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:00.828561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:39211 (local address 127.31.131.65:48295)
0507 02:08:00.828694 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0507 02:08:00.828707 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0507 02:08:00.828760 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:08:00.828902 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:08:00.828903 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:08:00.829099 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:08:00.829323 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:00.829326 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:00.829941 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:08:00.829941 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:08:00.830308 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:00.830309 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:00.830365 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:00.830829 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:08:00.830833 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:08:00.832416 (+ 1583us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.131.126:39211: Server oryx/127.31.131.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":64,"threads_started":1}
W20240507 02:08:00.832579 9043 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:08:00.828645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:48295 (local address 127.31.131.126:39211)
0507 02:08:00.828747 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:00.828750 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:00.828760 (+ 10us) server_negotiation.cc:407] Connection header received
0507 02:08:00.828820 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:00.828821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:00.828830 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:00.828881 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:08:00.829384 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:00.829884 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:08:00.830367 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:00.830469 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:00.832522 (+ 2053us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.131.65:48295: BlockingRecv error: recv got EOF from 127.31.131.65:48295 (error 108)
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":52,"threads_started":1}
May 07 02:08:00 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.66@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
May 07 02:08:00 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.66@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
I20240507 02:08:00.957342 9044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:00.953624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:39211 (local address 127.31.131.66:33179)
0507 02:08:00.953728 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0507 02:08:00.953738 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0507 02:08:00.953797 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:08:00.953909 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:08:00.953910 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:08:00.954074 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:08:00.954254 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:00.954255 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:00.954854 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:08:00.954855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:08:00.955226 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:00.955228 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:00.955277 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:00.955759 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:08:00.955763 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:08:00.957279 (+ 1516us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.131.126:39211: Server oryx/127.31.131.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":48,"threads_started":1}
W20240507 02:08:00.957425 9043 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:08:00.953683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.66:33179 (local address 127.31.131.126:39211)
0507 02:08:00.953747 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:00.953749 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:00.953794 (+ 45us) server_negotiation.cc:407] Connection header received
0507 02:08:00.953839 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:00.953839 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:00.953852 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:00.953900 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:08:00.954310 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:00.954797 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:08:00.955288 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:00.955402 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:00.957370 (+ 1968us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.131.66:33179: BlockingRecv error: recv got EOF from 127.31.131.66:33179 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
May 07 02:08:01 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.67@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
May 07 02:08:01 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.67@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
I20240507 02:08:01.077107 9045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:01.073278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:39211 (local address 127.31.131.67:56845)
0507 02:08:01.073412 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0507 02:08:01.073423 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0507 02:08:01.073494 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:08:01.073624 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:08:01.073625 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:08:01.073784 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:08:01.073972 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:01.073974 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:01.074616 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:08:01.074618 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:08:01.074988 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:01.074989 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:01.075047 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:01.075491 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:08:01.075494 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:08:01.077046 (+ 1552us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.131.126:39211: Server oryx/127.31.131.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":72,"threads_started":1}
W20240507 02:08:01.077194 9043 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:08:01.073348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.67:56845 (local address 127.31.131.126:39211)
0507 02:08:01.073416 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:01.073418 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:01.073484 (+ 66us) server_negotiation.cc:407] Connection header received
0507 02:08:01.073556 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:01.073558 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:01.073569 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:01.073609 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:08:01.074044 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:01.074520 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:08:01.075055 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:01.075206 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:01.077131 (+ 1925us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.131.67:56845: BlockingRecv error: recv got EOF from 127.31.131.67:56845 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
May 07 02:08:01 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.65@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
May 07 02:08:01 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.65@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
I20240507 02:08:01.836910 9046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:01.832936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:39211 (local address 127.31.131.65:46449)
0507 02:08:01.833065 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0507 02:08:01.833075 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0507 02:08:01.833122 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:08:01.833253 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:08:01.833254 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:08:01.833403 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:08:01.833559 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:01.833560 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:01.834165 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:08:01.834166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:08:01.834521 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:01.834522 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:01.834614 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:01.835053 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:08:01.835057 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:08:01.836853 (+ 1796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.131.126:39211: Server oryx/127.31.131.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":76,"threads_started":1}
W20240507 02:08:01.837010 9047 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:08:01.833031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:46449 (local address 127.31.131.126:39211)
0507 02:08:01.833149 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:01.833151 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:01.833159 (+ 8us) server_negotiation.cc:407] Connection header received
0507 02:08:01.833180 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:01.833180 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:01.833189 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:01.833242 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:08:01.833613 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:01.834110 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:08:01.834623 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:01.834741 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:01.836948 (+ 2207us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.131.65:46449: BlockingRecv error: recv got EOF from 127.31.131.65:46449 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":64,"threads_started":1}
May 07 02:08:01 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.66@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
May 07 02:08:01 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.66@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
I20240507 02:08:01.961439 9048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:01.957761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:39211 (local address 127.31.131.66:44757)
0507 02:08:01.957879 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0507 02:08:01.957888 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0507 02:08:01.957944 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:08:01.958053 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:08:01.958054 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:08:01.958208 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:08:01.958365 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:01.958366 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:01.958961 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:08:01.958961 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:08:01.959329 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:01.959330 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:01.959385 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:01.959826 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:08:01.959829 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:08:01.961382 (+ 1553us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.131.126:39211: Server oryx/127.31.131.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":69,"threads_started":1}
W20240507 02:08:01.961546 9047 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:08:01.957851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.66:44757 (local address 127.31.131.126:39211)
0507 02:08:01.957894 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:01.957896 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:01.957944 (+ 48us) server_negotiation.cc:407] Connection header received
0507 02:08:01.957994 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:01.957995 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:01.958006 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:01.958043 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:08:01.958418 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:01.958906 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:08:01.959387 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:01.959486 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:01.961484 (+ 1998us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.131.66:44757: BlockingRecv error: recv got EOF from 127.31.131.66:44757 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
May 07 02:08:02 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.67@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
May 07 02:08:02 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.67@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
I20240507 02:08:02.081419 9049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:02.077503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:39211 (local address 127.31.131.67:58669)
0507 02:08:02.077621 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0507 02:08:02.077631 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0507 02:08:02.077676 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:08:02.077785 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:08:02.077786 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:08:02.077931 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:08:02.078087 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:02.078089 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:02.078752 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:08:02.078752 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:08:02.079127 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:02.079128 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:02.079177 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:02.079674 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:08:02.079677 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:08:02.081362 (+ 1685us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.131.126:39211: Server oryx/127.31.131.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":58,"threads_started":1}
W20240507 02:08:02.081513 9047 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:08:02.077610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.67:58669 (local address 127.31.131.126:39211)
0507 02:08:02.077661 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:02.077662 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:02.077670 (+ 8us) server_negotiation.cc:407] Connection header received
0507 02:08:02.077718 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:02.077719 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:02.077729 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:02.077774 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:08:02.078144 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:02.078684 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:08:02.079179 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:02.079280 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:02.081449 (+ 2169us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.131.67:58669: BlockingRecv error: recv got EOF from 127.31.131.67:58669 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
May 07 02:08:02 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.65@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
May 07 02:08:02 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.65@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
I20240507 02:08:02.840967 9050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:02.837308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:39211 (local address 127.31.131.65:60519)
0507 02:08:02.837421 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0507 02:08:02.837430 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0507 02:08:02.837475 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:08:02.837596 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:08:02.837597 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:08:02.837754 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:08:02.837909 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:02.837911 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:02.838513 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:08:02.838513 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:08:02.838906 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:02.838908 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:02.838955 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:02.839388 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:08:02.839392 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:08:02.840911 (+ 1519us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.131.126:39211: Server oryx/127.31.131.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":63,"threads_started":1}
W20240507 02:08:02.841105 9051 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:08:02.837361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:60519 (local address 127.31.131.126:39211)
0507 02:08:02.837466 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:02.837467 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:02.837475 (+ 8us) server_negotiation.cc:407] Connection header received
0507 02:08:02.837531 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:02.837531 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:02.837540 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:02.837584 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:08:02.837964 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:02.838461 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:08:02.838955 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:02.839060 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:02.841033 (+ 1973us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.131.65:60519: BlockingRecv error: recv got EOF from 127.31.131.65:60519 (error 108)
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":56,"threads_started":1}
May 07 02:08:02 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.66@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
May 07 02:08:02 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.66@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
I20240507 02:08:02.965407 9052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:02.961778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:39211 (local address 127.31.131.66:43045)
0507 02:08:02.961884 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0507 02:08:02.961894 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0507 02:08:02.961948 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:08:02.962053 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:08:02.962054 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:08:02.962199 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:08:02.962351 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:02.962371 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:02.963046 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:08:02.963047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:08:02.963421 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:02.963423 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:02.963474 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:02.963887 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:08:02.963890 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:08:02.965352 (+ 1462us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.131.126:39211: Server oryx/127.31.131.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":50,"threads_started":1}
W20240507 02:08:02.965497 9051 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:08:02.961837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.66:43045 (local address 127.31.131.126:39211)
0507 02:08:02.961887 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:02.961889 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:02.961943 (+ 54us) server_negotiation.cc:407] Connection header received
0507 02:08:02.961986 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:02.961987 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:02.961998 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:02.962042 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:08:02.962437 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:02.962978 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:08:02.963476 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:02.963578 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:02.965442 (+ 1864us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.131.66:43045: BlockingRecv error: recv got EOF from 127.31.131.66:43045 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
May 07 02:08:03 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.67@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
May 07 02:08:03 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.67@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
I20240507 02:08:03.085455 9053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:03.081802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:39211 (local address 127.31.131.67:58291)
0507 02:08:03.081914 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0507 02:08:03.081923 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0507 02:08:03.081975 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:08:03.082085 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:08:03.082085 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:08:03.082230 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:08:03.082387 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:03.082388 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:03.082982 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:08:03.082983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:08:03.083363 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:03.083364 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:03.083417 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:03.083885 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:08:03.083889 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:08:03.085400 (+ 1511us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.131.126:39211: Server oryx/127.31.131.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":59,"threads_started":1}
W20240507 02:08:03.085541 9051 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:08:03.081860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.67:58291 (local address 127.31.131.126:39211)
0507 02:08:03.081914 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:03.081915 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:03.081972 (+ 57us) server_negotiation.cc:407] Connection header received
0507 02:08:03.082024 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:03.082024 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:03.082036 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:03.082071 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:08:03.082439 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:03.082925 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:08:03.083419 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:03.083516 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:03.085479 (+ 1963us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.131.67:58291: BlockingRecv error: recv got EOF from 127.31.131.67:58291 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
May 07 02:08:03 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.65@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
May 07 02:08:03 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.65@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
I20240507 02:08:03.845160 9054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:03.841386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:39211 (local address 127.31.131.65:55835)
0507 02:08:03.841525 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0507 02:08:03.841536 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0507 02:08:03.841595 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:08:03.841714 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:08:03.841715 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:08:03.841874 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:08:03.842034 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:03.842035 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:03.842653 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:08:03.842654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:08:03.843015 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:03.843016 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:03.843067 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:03.843567 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:08:03.843571 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:08:03.845103 (+ 1532us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.131.126:39211: Server oryx/127.31.131.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":84,"threads_started":1}
W20240507 02:08:03.845273 9055 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:08:03.841453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:55835 (local address 127.31.131.126:39211)
0507 02:08:03.841544 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:03.841545 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:03.841587 (+ 42us) server_negotiation.cc:407] Connection header received
0507 02:08:03.841641 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:03.841642 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:03.841650 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:03.841701 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:08:03.842088 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:03.842596 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:08:03.843068 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:03.843168 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:03.845205 (+ 2037us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.131.65:55835: BlockingRecv error: recv got EOF from 127.31.131.65:55835 (error 108)
Metrics: {"server-negotiator.queue_time_us":74,"thread_start_us":45,"threads_started":1}
May 07 02:08:03 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.66@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
May 07 02:08:03 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.66@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
I20240507 02:08:03.969540 9056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:03.965788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:39211 (local address 127.31.131.66:57031)
0507 02:08:03.965913 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0507 02:08:03.965923 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0507 02:08:03.965981 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:08:03.966097 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:08:03.966097 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:08:03.966255 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:08:03.966416 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:03.966417 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:03.967073 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:08:03.967074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:08:03.967448 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:03.967449 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:03.967507 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:03.967948 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:08:03.967952 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:08:03.969483 (+ 1531us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.131.126:39211: Server oryx/127.31.131.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":63,"threads_started":1}
W20240507 02:08:03.969630 9055 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:08:03.965848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.66:57031 (local address 127.31.131.126:39211)
0507 02:08:03.965907 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:03.965909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:03.965973 (+ 64us) server_negotiation.cc:407] Connection header received
0507 02:08:03.966029 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:03.966029 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:03.966041 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:03.966086 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:08:03.966489 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:03.967005 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:08:03.967521 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:03.967677 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:03.969566 (+ 1889us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.131.66:57031: BlockingRecv error: recv got EOF from 127.31.131.66:57031 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
May 07 02:08:04 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.67@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
May 07 02:08:04 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.67@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
I20240507 02:08:04.089563 9057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:04.085874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:39211 (local address 127.31.131.67:34107)
0507 02:08:04.086008 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0507 02:08:04.086018 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0507 02:08:04.086074 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:08:04.086183 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:08:04.086184 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:08:04.086334 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:08:04.086485 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:04.086486 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:04.087135 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:08:04.087136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:08:04.087493 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:04.087494 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:04.087544 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:04.088013 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:08:04.088016 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:08:04.089507 (+ 1491us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.131.126:39211: Server oryx/127.31.131.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":71,"threads_started":1}
W20240507 02:08:04.089672 9055 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:08:04.085948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.67:34107 (local address 127.31.131.126:39211)
0507 02:08:04.086017 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:04.086018 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:04.086069 (+ 51us) server_negotiation.cc:407] Connection header received
0507 02:08:04.086122 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:04.086123 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:04.086134 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:04.086170 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:08:04.086589 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:04.087071 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:08:04.087563 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:04.087687 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:04.089606 (+ 1919us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.131.67:34107: BlockingRecv error: recv got EOF from 127.31.131.67:34107 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
May 07 02:08:04 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.65@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
May 07 02:08:04 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.65@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
I20240507 02:08:04.849839 9058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:04.845554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:39211 (local address 127.31.131.65:36693)
0507 02:08:04.845678 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0507 02:08:04.845686 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0507 02:08:04.845727 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:08:04.845886 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:08:04.845887 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:08:04.846028 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:08:04.846180 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:04.846182 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:04.847066 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:08:04.847067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:08:04.847655 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:04.847658 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:04.847734 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:04.848160 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:08:04.848164 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:08:04.849786 (+ 1622us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.131.126:39211: Server oryx/127.31.131.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":52,"threads_started":1}
W20240507 02:08:04.849984 9059 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:08:04.845610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:36693 (local address 127.31.131.126:39211)
0507 02:08:04.845713 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:04.845714 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:04.845724 (+ 10us) server_negotiation.cc:407] Connection header received
0507 02:08:04.845797 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:04.845798 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:04.845810 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:04.845882 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:08:04.846253 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:04.846985 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:08:04.847730 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:04.847866 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:04.849917 (+ 2051us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.131.65:36693: BlockingRecv error: recv got EOF from 127.31.131.65:36693 (error 108)
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":50,"threads_started":1}
May 07 02:08:04 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.66@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
May 07 02:08:04 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.66@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
I20240507 02:08:04.973512 9060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:04.969938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:39211 (local address 127.31.131.66:57589)
0507 02:08:04.970072 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0507 02:08:04.970081 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0507 02:08:04.970128 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:08:04.970235 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:08:04.970236 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:08:04.970399 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:08:04.970577 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:04.970578 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:04.971142 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:08:04.971142 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:08:04.971525 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:04.971527 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:04.971572 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:04.972005 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:08:04.972008 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:08:04.973458 (+ 1450us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.131.126:39211: Server oryx/127.31.131.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":72,"threads_started":1}
W20240507 02:08:04.973608 9059 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:08:04.970064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.66:57589 (local address 127.31.131.126:39211)
0507 02:08:04.970116 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:04.970118 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:04.970126 (+ 8us) server_negotiation.cc:407] Connection header received
0507 02:08:04.970176 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:04.970177 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:04.970188 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:04.970224 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:08:04.970625 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:04.971091 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:08:04.971579 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:04.971696 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:04.973546 (+ 1850us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.131.66:57589: BlockingRecv error: recv got EOF from 127.31.131.66:57589 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
May 07 02:08:05 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.67@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
May 07 02:08:05 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.67@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
I20240507 02:08:05.093631 9061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:05.089961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:39211 (local address 127.31.131.67:50281)
0507 02:08:05.090075 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0507 02:08:05.090085 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0507 02:08:05.090146 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:08:05.090251 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:08:05.090251 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:08:05.090405 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:08:05.090586 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:05.090588 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:05.091143 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:08:05.091144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:08:05.091511 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:05.091512 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:05.091559 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:05.092009 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:08:05.092013 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:08:05.093560 (+ 1547us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.131.126:39211: Server oryx/127.31.131.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":53,"threads_started":1}
W20240507 02:08:05.093708 9059 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:08:05.090027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.67:50281 (local address 127.31.131.126:39211)
0507 02:08:05.090098 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:05.090100 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:05.090142 (+ 42us) server_negotiation.cc:407] Connection header received
0507 02:08:05.090184 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:05.090185 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:05.090196 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:05.090238 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:08:05.090643 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:05.091095 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:08:05.091558 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:05.091654 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:05.093657 (+ 2003us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.131.67:50281: BlockingRecv error: recv got EOF from 127.31.131.67:50281 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
May 07 02:08:05 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.65@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
May 07 02:08:05 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.65@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
I20240507 02:08:05.853952 9062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:05.850239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:39211 (local address 127.31.131.65:35931)
0507 02:08:05.850358 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0507 02:08:05.850368 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0507 02:08:05.850427 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:08:05.850532 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:08:05.850533 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:08:05.850717 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:08:05.850922 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:05.850925 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:05.851545 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:08:05.851545 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:08:05.851912 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:05.851913 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:05.851960 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:05.852400 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:08:05.852404 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:08:05.853880 (+ 1476us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.131.126:39211: Server oryx/127.31.131.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":56,"threads_started":1}
W20240507 02:08:05.854038 9063 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:08:05.850308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:35931 (local address 127.31.131.126:39211)
0507 02:08:05.850433 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:05.850434 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:05.850444 (+ 10us) server_negotiation.cc:407] Connection header received
0507 02:08:05.850464 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:05.850464 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:05.850473 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:05.850522 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:08:05.850986 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:05.851488 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:08:05.851962 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:05.852064 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:05.853960 (+ 1896us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.131.65:35931: BlockingRecv error: recv got EOF from 127.31.131.65:35931 (error 108)
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":64,"threads_started":1}
May 07 02:08:05 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.66@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
May 07 02:08:05 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.66@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
I20240507 02:08:05.977916 9064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:05.973945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:39211 (local address 127.31.131.66:38421)
0507 02:08:05.974069 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0507 02:08:05.974078 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0507 02:08:05.974136 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:08:05.974241 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:08:05.974242 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:08:05.974409 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:08:05.974586 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:05.974588 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:05.975191 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:08:05.975192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:08:05.975627 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:05.975629 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:05.975688 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:05.976126 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:08:05.976129 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:08:05.977841 (+ 1712us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.131.126:39211: Server oryx/127.31.131.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":69,"threads_started":1}
W20240507 02:08:05.977962 9063 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:08:05.974013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.66:38421 (local address 127.31.131.126:39211)
0507 02:08:05.974061 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:05.974063 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:05.974129 (+ 66us) server_negotiation.cc:407] Connection header received
0507 02:08:05.974176 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:05.974177 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:05.974186 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:05.974230 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:08:05.974648 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:05.975113 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:08:05.975687 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:05.975784 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:05.977919 (+ 2135us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.131.66:38421: BlockingRecv error: recv got EOF from 127.31.131.66:38421 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
May 07 02:08:06 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.67@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
May 07 02:08:06 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.67@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
I20240507 02:08:06.097980 9065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:06.094067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:39211 (local address 127.31.131.67:56643)
0507 02:08:06.094183 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0507 02:08:06.094192 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0507 02:08:06.094236 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:08:06.094367 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:08:06.094368 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:08:06.094529 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:08:06.094714 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:06.094716 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:06.095312 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:08:06.095313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:08:06.095757 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:06.095759 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:06.095812 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:06.096242 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:08:06.096246 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:08:06.097897 (+ 1651us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.131.126:39211: Server oryx/127.31.131.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":63,"threads_started":1}
W20240507 02:08:06.098032 9063 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:08:06.094148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.67:56643 (local address 127.31.131.126:39211)
0507 02:08:06.094217 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:06.094219 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:06.094228 (+ 9us) server_negotiation.cc:407] Connection header received
0507 02:08:06.094293 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:06.094293 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:06.094305 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:06.094357 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:08:06.094774 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:06.095239 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:08:06.095818 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:06.095943 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:06.097983 (+ 2040us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.131.67:56643: BlockingRecv error: recv got EOF from 127.31.131.67:56643 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
May 07 02:08:06 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.65@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
May 07 02:08:06 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.65@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
I20240507 02:08:06.858208 9066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:06.854360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:39211 (local address 127.31.131.65:38819)
0507 02:08:06.854499 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0507 02:08:06.854509 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0507 02:08:06.854586 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:08:06.854712 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:08:06.854713 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:08:06.854877 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:08:06.855036 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:06.855038 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:06.855643 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:08:06.855644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:08:06.856017 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:06.856019 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:06.856075 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:06.856564 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:08:06.856568 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:08:06.858135 (+ 1567us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.131.126:39211: Server oryx/127.31.131.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":82,"threads_started":1}
W20240507 02:08:06.858305 9067 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:08:06.854426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:38819 (local address 127.31.131.126:39211)
0507 02:08:06.854532 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:06.854534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:06.854543 (+ 9us) server_negotiation.cc:407] Connection header received
0507 02:08:06.854637 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:06.854638 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:06.854646 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:06.854697 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:08:06.855091 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:06.855587 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:08:06.856071 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:06.856183 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:06.858248 (+ 2065us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.131.65:38819: BlockingRecv error: recv got EOF from 127.31.131.65:38819 (error 108)
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":52,"threads_started":1}
May 07 02:08:06 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.66@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
May 07 02:08:06 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.66@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
I20240507 02:08:06.981958 9068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:06.978302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:39211 (local address 127.31.131.66:45291)
0507 02:08:06.978440 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0507 02:08:06.978450 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0507 02:08:06.978510 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:08:06.978659 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:08:06.978659 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:08:06.978813 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:08:06.978960 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:06.978962 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:06.979559 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:08:06.979560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:08:06.979915 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:06.979916 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:06.979971 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:06.980408 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:08:06.980411 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:08:06.981885 (+ 1474us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.131.126:39211: Server oryx/127.31.131.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":69,"threads_started":1}
W20240507 02:08:06.982051 9067 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:08:06.978368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.66:45291 (local address 127.31.131.126:39211)
0507 02:08:06.978429 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:06.978431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:06.978498 (+ 67us) server_negotiation.cc:407] Connection header received
0507 02:08:06.978589 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:06.978590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:06.978603 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:06.978650 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:08:06.979028 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:06.979493 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:08:06.979978 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:06.980134 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:06.982000 (+ 1866us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.131.66:45291: BlockingRecv error: recv got EOF from 127.31.131.66:45291 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
May 07 02:08:07 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.67@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
May 07 02:08:07 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.67@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
I20240507 02:08:07.102151 9069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:07.098376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:39211 (local address 127.31.131.67:44511)
0507 02:08:07.098518 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0507 02:08:07.098527 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0507 02:08:07.098597 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:08:07.098713 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:08:07.098713 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:08:07.098877 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:08:07.099024 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:07.099026 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:07.099715 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:08:07.099716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:08:07.100080 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:07.100082 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:07.100142 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:07.100580 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:08:07.100583 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:08:07.102090 (+ 1507us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.131.126:39211: Server oryx/127.31.131.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":77,"threads_started":1}
W20240507 02:08:07.102253 9067 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:08:07.098431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.67:44511 (local address 127.31.131.126:39211)
0507 02:08:07.098496 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:07.098498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:07.098592 (+ 94us) server_negotiation.cc:407] Connection header received
0507 02:08:07.098644 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:07.098645 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:07.098658 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:07.098704 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:08:07.099094 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:07.099646 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:08:07.100150 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:07.100315 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:07.102192 (+ 1877us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.131.67:44511: BlockingRecv error: recv got EOF from 127.31.131.67:44511 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
May 07 02:08:07 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.65@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
May 07 02:08:07 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.65@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
I20240507 02:08:07.862486 9070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:07.858664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:39211 (local address 127.31.131.65:51255)
0507 02:08:07.858787 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0507 02:08:07.858796 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0507 02:08:07.858847 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:08:07.858968 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:08:07.858969 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:08:07.859115 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:08:07.859266 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:07.859268 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:07.859894 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:08:07.859894 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:08:07.860250 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:07.860250 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:07.860296 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:07.860734 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:08:07.860738 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:08:07.862432 (+ 1694us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.131.126:39211: Server oryx/127.31.131.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":56,"threads_started":1}
W20240507 02:08:07.862634 9071 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:08:07.858758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:51255 (local address 127.31.131.126:39211)
0507 02:08:07.858860 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:07.858862 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:07.858874 (+ 12us) server_negotiation.cc:407] Connection header received
0507 02:08:07.858893 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:07.858894 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:07.858903 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:07.858956 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:08:07.859323 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:07.859837 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:08:07.860301 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:07.860411 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:07.862525 (+ 2114us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.131.65:51255: BlockingRecv error: recv got EOF from 127.31.131.65:51255 (error 108)
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":53,"threads_started":1}
May 07 02:08:07 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.66@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
May 07 02:08:07 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.66@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
I20240507 02:08:07.986311 9072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:07.982388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:39211 (local address 127.31.131.66:53297)
0507 02:08:07.982532 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0507 02:08:07.982542 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0507 02:08:07.982629 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:08:07.982745 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:08:07.982745 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:08:07.982906 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:08:07.983056 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:07.983057 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:07.983830 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:08:07.983831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:08:07.984191 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:07.984192 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:07.984250 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:07.984702 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:08:07.984705 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:08:07.986236 (+ 1531us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.131.126:39211: Server oryx/127.31.131.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":88,"threads_started":1}
W20240507 02:08:07.986388 9071 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:08:07.982458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.66:53297 (local address 127.31.131.126:39211)
0507 02:08:07.982531 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:07.982533 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:07.982622 (+ 89us) server_negotiation.cc:407] Connection header received
0507 02:08:07.982676 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:07.982677 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:07.982689 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:07.982735 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:08:07.983132 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:07.983761 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:08:07.984266 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:07.984430 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:07.986337 (+ 1907us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.131.66:53297: BlockingRecv error: recv got EOF from 127.31.131.66:53297 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
May 07 02:08:08 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.67@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
May 07 02:08:08 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.67@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
I20240507 02:08:08.106433 9073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:08.102520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:39211 (local address 127.31.131.67:57655)
0507 02:08:08.102660 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0507 02:08:08.102670 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0507 02:08:08.102729 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:08:08.102849 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:08:08.102850 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:08:08.103013 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:08:08.103161 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:08.103163 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:08.103786 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:08:08.103787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:08:08.104156 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:08.104157 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:08.104289 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:08.104787 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:08:08.104800 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:08:08.106359 (+ 1559us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.131.126:39211: Server oryx/127.31.131.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":71,"threads_started":1}
W20240507 02:08:08.106503 9071 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:08:08.102600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.67:57655 (local address 127.31.131.126:39211)
0507 02:08:08.102667 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:08.102669 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:08.102724 (+ 55us) server_negotiation.cc:407] Connection header received
0507 02:08:08.102776 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:08.102777 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:08.102791 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:08.102835 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:08:08.103235 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:08.103716 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:08:08.104301 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:08.104491 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:08.106452 (+ 1961us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.131.67:57655: BlockingRecv error: recv got EOF from 127.31.131.67:57655 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
May 07 02:08:08 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.65@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
May 07 02:08:08 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.65@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
I20240507 02:08:08.866654 9074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:08.862886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:39211 (local address 127.31.131.65:56527)
0507 02:08:08.862998 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0507 02:08:08.863007 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0507 02:08:08.863051 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:08:08.863163 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:08:08.863163 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:08:08.863320 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:08:08.863473 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:08.863475 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:08.864086 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:08:08.864087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:08:08.864466 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:08.864468 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:08.864525 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:08.864960 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:08:08.864963 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:08:08.866598 (+ 1635us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.131.126:39211: Server oryx/127.31.131.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":55,"threads_started":1}
W20240507 02:08:08.866751 9075 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:08:08.862947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:56527 (local address 127.31.131.126:39211)
0507 02:08:08.863046 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:08.863047 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:08.863055 (+ 8us) server_negotiation.cc:407] Connection header received
0507 02:08:08.863091 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:08.863092 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:08.863100 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:08.863153 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:08:08.863527 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:08.864009 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:08:08.864523 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:08.864623 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:08.866692 (+ 2069us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.131.65:56527: BlockingRecv error: recv got EOF from 127.31.131.65:56527 (error 108)
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":50,"threads_started":1}
May 07 02:08:08 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.66@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
May 07 02:08:08 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.66@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
I20240507 02:08:08.994489 9076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:08.986767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:39211 (local address 127.31.131.66:47025)
0507 02:08:08.986887 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0507 02:08:08.986896 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0507 02:08:08.986953 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:08:08.987062 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:08:08.987063 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:08:08.987209 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:08:08.987362 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:08.987363 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:08.987936 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:08:08.987936 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:08:08.988286 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:08.988287 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:08.988331 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:08.992864 (+ 4533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:08:08.992868 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:08:08.994423 (+ 1555us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.131.126:39211: Server oryx/127.31.131.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":60,"threads_started":1}
W20240507 02:08:08.994639 9075 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:08:08.986825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.66:47025 (local address 127.31.131.126:39211)
0507 02:08:08.986891 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:08.986893 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:08.986949 (+ 56us) server_negotiation.cc:407] Connection header received
0507 02:08:08.987001 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:08.987002 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:08.987013 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:08.987049 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:08:08.987416 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:08.987884 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:08:08.988334 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:08.988434 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:08.994543 (+ 6109us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.131.66:47025: BlockingRecv error: recv got EOF from 127.31.131.66:47025 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
May 07 02:08:09 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.67@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
May 07 02:08:09 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.67@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
I20240507 02:08:09.110438 9077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:09.106851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:39211 (local address 127.31.131.67:53633)
0507 02:08:09.106972 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0507 02:08:09.106981 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0507 02:08:09.107034 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:08:09.107142 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:08:09.107142 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:08:09.107284 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:08:09.107436 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:09.107437 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:09.107995 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:08:09.107996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:08:09.108350 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:09.108351 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:09.108397 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:09.108809 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:08:09.108813 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:08:09.110377 (+ 1564us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.131.126:39211: Server oryx/127.31.131.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":57,"threads_started":1}
W20240507 02:08:09.110524 9075 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:08:09.106906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.67:53633 (local address 127.31.131.126:39211)
0507 02:08:09.106969 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:09.106971 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:09.107030 (+ 59us) server_negotiation.cc:407] Connection header received
0507 02:08:09.107081 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:09.107082 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:09.107093 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:09.107128 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:08:09.107487 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:09.107947 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:08:09.108398 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:09.108494 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:09.110462 (+ 1968us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.131.67:53633: BlockingRecv error: recv got EOF from 127.31.131.67:53633 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
May 07 02:08:09 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.65@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
May 07 02:08:09 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.65@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
I20240507 02:08:09.875849 9078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:09.867035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:39211 (local address 127.31.131.65:40293)
0507 02:08:09.867154 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0507 02:08:09.867165 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0507 02:08:09.867215 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:08:09.867333 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:08:09.867334 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:08:09.867479 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:08:09.867629 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:09.867630 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:09.868208 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:08:09.868209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:08:09.868593 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:09.868594 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:09.868641 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:09.874226 (+ 5585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:08:09.874231 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:08:09.875783 (+ 1552us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.131.126:39211: Server oryx/127.31.131.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":56,"threads_started":1}
W20240507 02:08:09.875937 9079 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:08:09.867085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:40293 (local address 127.31.131.126:39211)
0507 02:08:09.867202 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:09.867204 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:09.867212 (+ 8us) server_negotiation.cc:407] Connection header received
0507 02:08:09.867263 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:09.867264 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:09.867273 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:09.867332 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:08:09.867680 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:09.868158 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:08:09.868644 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:09.868748 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:09.875876 (+ 7128us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.131.65:40293: BlockingRecv error: recv got EOF from 127.31.131.65:40293 (error 108)
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":60,"threads_started":1}
May 07 02:08:09 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.66@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
May 07 02:08:09 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.66@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
I20240507 02:08:09.999047 9080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:09.994957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:39211 (local address 127.31.131.66:35657)
0507 02:08:09.995095 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0507 02:08:09.995105 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0507 02:08:09.995169 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:08:09.995285 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:08:09.995286 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:08:09.995449 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:08:09.995609 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:09.995611 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:09.996251 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:08:09.996252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:08:09.996631 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:09.996632 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:09.996690 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:09.997129 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:08:09.997133 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:08:09.998975 (+ 1842us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.131.126:39211: Server oryx/127.31.131.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":79,"threads_started":1}
W20240507 02:08:09.999150 9079 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:08:09.995014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.66:35657 (local address 127.31.131.126:39211)
0507 02:08:09.995067 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:09.995069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:09.995159 (+ 90us) server_negotiation.cc:407] Connection header received
0507 02:08:09.995220 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:09.995221 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:09.995232 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:09.995274 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:08:09.995725 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:09.996193 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:08:09.996688 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:09.996830 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:09.999093 (+ 2263us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.131.66:35657: BlockingRecv error: recv got EOF from 127.31.131.66:35657 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
May 07 02:08:10 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.67@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
May 07 02:08:10 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.67@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
I20240507 02:08:10.114435 9081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:10.110855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:39211 (local address 127.31.131.67:44541)
0507 02:08:10.110966 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0507 02:08:10.110975 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0507 02:08:10.111037 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:08:10.111163 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:08:10.111163 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:08:10.111316 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:08:10.111477 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:10.111479 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:10.112076 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:08:10.112077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:08:10.112442 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:10.112443 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:10.112500 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:10.112937 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:08:10.112941 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:08:10.114362 (+ 1421us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.131.126:39211: Server oryx/127.31.131.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":56,"threads_started":1}
W20240507 02:08:10.114492 9079 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:08:10.110967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.67:44541 (local address 127.31.131.126:39211)
0507 02:08:10.111031 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:10.111033 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:10.111041 (+ 8us) server_negotiation.cc:407] Connection header received
0507 02:08:10.111094 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:10.111094 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:10.111107 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:10.111150 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:08:10.111545 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:10.112012 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:08:10.112508 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:10.112665 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:10.114444 (+ 1779us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.131.67:44541: BlockingRecv error: recv got EOF from 127.31.131.67:44541 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
May 07 02:08:10 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.65@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
May 07 02:08:10 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.65@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
I20240507 02:08:10.880146 9082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:10.876232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:39211 (local address 127.31.131.65:51845)
0507 02:08:10.876342 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0507 02:08:10.876351 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0507 02:08:10.876394 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:08:10.876511 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:08:10.876511 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:08:10.876656 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:08:10.876808 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:10.876810 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:10.877603 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:08:10.877604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:08:10.877989 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:10.877990 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:10.878041 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:10.878512 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:08:10.878516 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:08:10.880082 (+ 1566us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.131.126:39211: Server oryx/127.31.131.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":55,"threads_started":1}
W20240507 02:08:10.880240 9083 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:08:10.876288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:51845 (local address 127.31.131.126:39211)
0507 02:08:10.876386 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:10.876388 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:10.876395 (+ 7us) server_negotiation.cc:407] Connection header received
0507 02:08:10.876447 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:10.876447 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:10.876456 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:10.876501 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:08:10.876863 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:10.877542 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:08:10.878041 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:10.878146 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:10.880174 (+ 2028us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.131.65:51845: BlockingRecv error: recv got EOF from 127.31.131.65:51845 (error 108)
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":50,"threads_started":1}
May 07 02:08:10 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.66@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
May 07 02:08:11 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.66@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
I20240507 02:08:11.003100 9084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:10.999532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:39211 (local address 127.31.131.66:51953)
0507 02:08:10.999639 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0507 02:08:10.999648 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0507 02:08:10.999699 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:08:10.999806 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:08:10.999806 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:08:10.999947 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:08:11.000096 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:11.000097 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:11.000671 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:08:11.000671 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:08:11.001021 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:11.001022 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:11.001065 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:11.001488 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:08:11.001491 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:08:11.003044 (+ 1553us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.131.126:39211: Server oryx/127.31.131.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":52,"threads_started":1}
W20240507 02:08:11.003198 9083 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:08:10.999600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.66:51953 (local address 127.31.131.126:39211)
0507 02:08:10.999649 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:10.999650 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:10.999698 (+ 48us) server_negotiation.cc:407] Connection header received
0507 02:08:10.999738 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:10.999738 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:10.999749 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:10.999797 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:08:11.000150 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:11.000621 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:08:11.001071 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:11.001171 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:11.003133 (+ 1962us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.131.66:51953: BlockingRecv error: recv got EOF from 127.31.131.66:51953 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
May 07 02:08:11 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.67@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
May 07 02:08:11 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.67@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
I20240507 02:08:11.118604 9085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:11.114867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:39211 (local address 127.31.131.67:38575)
0507 02:08:11.114988 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0507 02:08:11.114997 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0507 02:08:11.115050 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:08:11.115157 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:08:11.115158 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:08:11.115324 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:08:11.115484 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:11.115486 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:11.116079 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:08:11.116079 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:08:11.116486 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:11.116487 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:11.116536 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:11.116987 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:08:11.116991 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:08:11.118478 (+ 1487us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.131.126:39211: Server oryx/127.31.131.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":82,"threads_started":1}
W20240507 02:08:11.118664 9083 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:08:11.114917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.67:38575 (local address 127.31.131.126:39211)
0507 02:08:11.114973 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:11.114974 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:11.115049 (+ 75us) server_negotiation.cc:407] Connection header received
0507 02:08:11.115090 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:11.115090 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:11.115103 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:11.115152 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:08:11.115545 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:11.116019 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:08:11.116553 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:11.116688 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:11.118609 (+ 1921us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.131.67:38575: BlockingRecv error: recv got EOF from 127.31.131.67:38575 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
May 07 02:08:11 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.65@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
May 07 02:08:11 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.65@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
I20240507 02:08:11.884222 9086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:11.880516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:39211 (local address 127.31.131.65:54783)
0507 02:08:11.880634 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0507 02:08:11.880650 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0507 02:08:11.880698 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:08:11.880809 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:08:11.880810 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:08:11.880957 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:08:11.881108 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:11.881110 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:11.881690 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:08:11.881690 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:08:11.882040 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:11.882041 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:11.882086 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:11.882535 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:08:11.882539 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:08:11.884164 (+ 1625us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.131.126:39211: Server oryx/127.31.131.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":57,"threads_started":1}
W20240507 02:08:11.884328 9087 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:08:11.880566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:54783 (local address 127.31.131.126:39211)
0507 02:08:11.880682 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:11.880683 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:11.880691 (+ 8us) server_negotiation.cc:407] Connection header received
0507 02:08:11.880745 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:11.880746 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:11.880755 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:11.880796 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:08:11.881162 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:11.881640 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:08:11.882091 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:11.882191 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:11.884257 (+ 2066us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.131.65:54783: BlockingRecv error: recv got EOF from 127.31.131.65:54783 (error 108)
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":63,"threads_started":1}
May 07 02:08:12 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.66@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
May 07 02:08:12 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.66@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
I20240507 02:08:12.007294 9088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:12.003498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:39211 (local address 127.31.131.66:33695)
0507 02:08:12.003609 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0507 02:08:12.003617 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0507 02:08:12.003676 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:08:12.003787 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:08:12.003787 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:08:12.003937 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:08:12.004087 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:12.004089 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:12.004653 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:08:12.004653 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:08:12.005005 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:12.005006 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:12.005052 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:12.005493 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:08:12.005498 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:08:12.007221 (+ 1723us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.131.126:39211: Server oryx/127.31.131.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":61,"threads_started":1}
W20240507 02:08:12.007411 9087 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:08:12.003547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.66:33695 (local address 127.31.131.126:39211)
0507 02:08:12.003596 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:12.003598 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:12.003667 (+ 69us) server_negotiation.cc:407] Connection header received
0507 02:08:12.003720 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:12.003721 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:12.003732 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:12.003779 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:08:12.004140 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:12.004602 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:08:12.005054 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:12.005149 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:12.007347 (+ 2198us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.131.66:33695: BlockingRecv error: recv got EOF from 127.31.131.66:33695 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
May 07 02:08:12 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.67@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
May 07 02:08:12 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.67@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
I20240507 02:08:12.122618 9089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:12.119002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:39211 (local address 127.31.131.67:38853)
0507 02:08:12.119130 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0507 02:08:12.119138 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0507 02:08:12.119190 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:08:12.119291 (+ 101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:08:12.119292 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:08:12.119436 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:08:12.119584 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:12.119586 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:12.120142 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:08:12.120143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:08:12.120500 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:12.120501 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:12.120547 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:12.120977 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:08:12.120980 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:08:12.122523 (+ 1543us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.131.126:39211: Server oryx/127.31.131.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":80,"threads_started":1}
W20240507 02:08:12.122771 9087 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:08:12.119060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.67:38853 (local address 127.31.131.126:39211)
0507 02:08:12.119116 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:12.119117 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:12.119188 (+ 71us) server_negotiation.cc:407] Connection header received
0507 02:08:12.119226 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:12.119227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:12.119237 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:12.119281 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:08:12.119636 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:12.120093 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:08:12.120551 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:12.120645 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:12.122673 (+ 2028us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.131.67:38853: BlockingRecv error: recv got EOF from 127.31.131.67:38853 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
May 07 02:08:12 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.65@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
May 07 02:08:12 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.65@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
I20240507 02:08:12.888396 9090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:12.884652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:39211 (local address 127.31.131.65:40823)
0507 02:08:12.884767 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0507 02:08:12.884777 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0507 02:08:12.884829 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:08:12.884943 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:08:12.884944 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:08:12.885097 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:08:12.885252 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:12.885254 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:12.885861 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:08:12.885862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:08:12.886220 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:12.886221 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:12.886273 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:12.886790 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:08:12.886794 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:08:12.888329 (+ 1535us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.131.126:39211: Server oryx/127.31.131.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":63,"threads_started":1}
W20240507 02:08:12.888494 9091 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:08:12.884722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:40823 (local address 127.31.131.126:39211)
0507 02:08:12.884824 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:12.884825 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:12.884834 (+ 9us) server_negotiation.cc:407] Connection header received
0507 02:08:12.884869 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:12.884869 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:12.884878 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:12.884931 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:08:12.885304 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:12.885806 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:08:12.886274 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:12.886377 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:12.888426 (+ 2049us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.131.65:40823: BlockingRecv error: recv got EOF from 127.31.131.65:40823 (error 108)
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":47,"threads_started":1}
May 07 02:08:13 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.66@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
May 07 02:08:13 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.66@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
I20240507 02:08:13.011579 9092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:13.007705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:39211 (local address 127.31.131.66:44169)
0507 02:08:13.007838 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0507 02:08:13.007848 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0507 02:08:13.007919 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:08:13.008043 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:08:13.008044 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:08:13.008207 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:08:13.008356 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:13.008358 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:13.009070 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:08:13.009073 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:08:13.009519 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:13.009521 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:13.009578 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:13.010029 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:08:13.010033 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:08:13.011510 (+ 1477us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.131.126:39211: Server oryx/127.31.131.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":61,"threads_started":1}
W20240507 02:08:13.011646 9091 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:08:13.007800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.66:44169 (local address 127.31.131.126:39211)
0507 02:08:13.007843 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:13.007845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:13.007911 (+ 66us) server_negotiation.cc:407] Connection header received
0507 02:08:13.007973 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:13.007974 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:13.007986 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:13.008032 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:08:13.008430 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:13.008951 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:08:13.009583 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:13.009702 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:13.011595 (+ 1893us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.131.66:44169: BlockingRecv error: recv got EOF from 127.31.131.66:44169 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
May 07 02:08:13 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.67@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
May 07 02:08:13 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.67@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
I20240507 02:08:13.126717 9093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:13.122981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:39211 (local address 127.31.131.67:58809)
0507 02:08:13.123108 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0507 02:08:13.123117 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0507 02:08:13.123171 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:08:13.123277 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:08:13.123278 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:08:13.123420 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:08:13.123569 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:13.123571 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:13.124126 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:08:13.124127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:08:13.124478 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:13.124479 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:13.124525 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:13.124964 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:08:13.124968 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:08:13.126641 (+ 1673us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.131.126:39211: Server oryx/127.31.131.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":61,"threads_started":1}
W20240507 02:08:13.126791 9091 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:08:13.123039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.67:58809 (local address 127.31.131.126:39211)
0507 02:08:13.123103 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:13.123105 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:13.123169 (+ 64us) server_negotiation.cc:407] Connection header received
0507 02:08:13.123210 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:13.123211 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:13.123223 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:13.123265 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:08:13.123621 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:13.124079 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:08:13.124538 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:13.124644 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:13.126740 (+ 2096us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.131.67:58809: BlockingRecv error: recv got EOF from 127.31.131.67:58809 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
May 07 02:08:13 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.65@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
May 07 02:08:13 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.65@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
I20240507 02:08:13.892544 9094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:13.888798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:39211 (local address 127.31.131.65:33751)
0507 02:08:13.888917 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0507 02:08:13.888926 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0507 02:08:13.888980 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:08:13.889096 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:08:13.889097 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:08:13.889253 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:08:13.889427 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:13.889428 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:13.890033 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:08:13.890034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:08:13.890418 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:13.890420 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:13.890478 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:13.890929 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:08:13.890933 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:08:13.892472 (+ 1539us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.131.126:39211: Server oryx/127.31.131.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":64,"threads_started":1}
W20240507 02:08:13.892632 9095 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:08:13.888886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:33751 (local address 127.31.131.126:39211)
0507 02:08:13.888986 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:13.888988 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:13.889013 (+ 25us) server_negotiation.cc:407] Connection header received
0507 02:08:13.889030 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:13.889031 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:13.889039 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:13.889082 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:08:13.889484 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:13.889971 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:08:13.890475 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:13.890614 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:13.892575 (+ 1961us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.131.65:33751: BlockingRecv error: recv got EOF from 127.31.131.65:33751 (error 108)
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":45,"threads_started":1}
May 07 02:08:14 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.66@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
May 07 02:08:14 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.66@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
I20240507 02:08:14.015682 9096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:14.011990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:39211 (local address 127.31.131.66:38045)
0507 02:08:14.012157 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0507 02:08:14.012168 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0507 02:08:14.012225 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:08:14.012321 (+ 96us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:08:14.012322 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:08:14.012478 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:08:14.012630 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:14.012631 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:14.013218 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:08:14.013219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:08:14.013583 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:14.013584 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:14.013630 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:14.014060 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:08:14.014063 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:08:14.015629 (+ 1566us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.131.126:39211: Server oryx/127.31.131.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":66,"threads_started":1}
W20240507 02:08:14.015780 9095 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:08:14.012033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.66:38045 (local address 127.31.131.126:39211)
0507 02:08:14.012079 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:14.012081 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:14.012218 (+ 137us) server_negotiation.cc:407] Connection header received
0507 02:08:14.012263 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:14.012263 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:14.012275 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:14.012317 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:08:14.012690 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:14.013160 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:08:14.013644 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:14.013754 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:14.015715 (+ 1961us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.131.66:38045: BlockingRecv error: recv got EOF from 127.31.131.66:38045 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
May 07 02:08:14 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.67@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
May 07 02:08:14 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.67@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
I20240507 02:08:14.130873 9097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:14.127093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:39211 (local address 127.31.131.67:38433)
0507 02:08:14.127222 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0507 02:08:14.127230 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0507 02:08:14.127283 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:08:14.127395 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:08:14.127395 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:08:14.127538 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:08:14.127709 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:14.127711 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:14.128438 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:08:14.128438 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:08:14.128803 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:14.128805 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:14.128853 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:14.129283 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:08:14.129287 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:08:14.130794 (+ 1507us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.131.126:39211: Server oryx/127.31.131.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":71,"threads_started":1}
W20240507 02:08:14.130939 9095 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:08:14.127154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.67:38433 (local address 127.31.131.126:39211)
0507 02:08:14.127206 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:14.127208 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:14.127278 (+ 70us) server_negotiation.cc:407] Connection header received
0507 02:08:14.127320 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:14.127321 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:14.127332 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:14.127376 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:08:14.127778 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:14.128357 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:08:14.128856 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:14.128958 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:14.130889 (+ 1931us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.131.67:38433: BlockingRecv error: recv got EOF from 127.31.131.67:38433 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
May 07 02:08:14 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.65@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
May 07 02:08:14 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.65@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
I20240507 02:08:14.900761 9098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:14.893037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:39211 (local address 127.31.131.65:46709)
0507 02:08:14.893205 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0507 02:08:14.893216 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0507 02:08:14.893269 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:08:14.893387 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:08:14.893388 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:08:14.893532 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:08:14.893685 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:14.893686 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:14.894316 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:08:14.894316 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:08:14.894731 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:14.894733 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:14.894782 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:14.899074 (+ 4292us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:08:14.899078 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:08:14.900686 (+ 1608us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.131.126:39211: Server oryx/127.31.131.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":88,"threads_started":1}
W20240507 02:08:14.900868 9099 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:08:14.893129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:46709 (local address 127.31.131.126:39211)
0507 02:08:14.893265 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:14.893267 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:14.893276 (+ 9us) server_negotiation.cc:407] Connection header received
0507 02:08:14.893302 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:14.893303 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:14.893314 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:14.893378 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:08:14.893741 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:14.894258 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:08:14.894787 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:14.894905 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:14.900809 (+ 5904us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.131.65:46709: BlockingRecv error: recv got EOF from 127.31.131.65:46709 (error 108)
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":75,"threads_started":1}
May 07 02:08:15 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.66@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
May 07 02:08:15 dist-test-slave-v0bg krb5kdc[8036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.131.66@KRBTEST.COM for oryx/127.31.131.126@KRBTEST.COM, Server not found in Kerberos database
I20240507 02:08:15.019768 9100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:15.016059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:39211 (local address 127.31.131.66:51993)
0507 02:08:15.016193 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0507 02:08:15.016203 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0507 02:08:15.016257 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:08:15.016393 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:08:15.016393 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:08:15.016541 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:08:15.016692 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:15.016694 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:15.017282 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:08:15.017283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:08:15.017655 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:15.017656 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:15.017708 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:15.018154 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:08:15.018158 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:08:15.019712 (+ 1554us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.131.126:39211: Server oryx/127.31.131.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":78,"threads_started":1}
W20240507 02:08:15.019863 9099 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:08:15.016104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.66:51993 (local address 127.31.131.126:39211)
0507 02:08:15.016180 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:15.016181 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:15.016255 (+ 74us) server_negotiation.cc:407] Connection header received
0507 02:08:15.016316 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:15.016317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:15.016328 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:15.016385 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:08:15.016748 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:15.017226 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:08:15.017713 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:15.017815 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:15.019797 (+ 1982us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.131.66:51993: BlockingRecv error: recv got EOF from 127.31.131.66:51993 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20240507 02:08:15.075163 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 8623
I20240507 02:08:15.079083 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 8776
I20240507 02:08:15.082727 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 8908
I20240507 02:08:15.086414 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 8552
2024-05-07T02:08:15Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17187 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2024-05-07T02:08:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-07T02:08:15Z Disabled control of system clock
I20240507 02:08:15.109737 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715047604443657-32269-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715047604443657-32269-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715047604443657-32269-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:37637
--webserver_interface=127.31.131.126
--webserver_port=0
--builtin_ntp_servers=127.31.131.84:41833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.131.126:37637
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20240507 02:08:15.183533 9104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:08:15.183683 9104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:08:15.183710 9104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:08:15.184931 9104 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240507 02:08:15.184969 9104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:08:15.184985 9104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:08:15.185000 9104 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240507 02:08:15.185014 9104 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240507 02:08:15.186384 9104 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:41833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715047604443657-32269-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715047604443657-32269-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.131.126:37637
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:37637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.131.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715047604443657-32269-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:08:15.186583 9104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:08:15.186784 9104 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:08:15.190222 9109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:08:15.190209 9110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:08:15.190375 9112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:08:15.190343 9104 server_base.cc:1034] running on GCE node
I20240507 02:08:15.190702 9104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:08:15.190901 9104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:08:15.192024 9104 hybrid_clock.cc:648] HybridClock initialized: now 1715047695192010 us; error 27 us; skew 500 ppm
I20240507 02:08:15.193042 9104 webserver.cc:460] Webserver started at http://127.31.131.126:39933/ using document root <none> and password file <none>
I20240507 02:08:15.193193 9104 fs_manager.cc:359] Metadata directory not provided
I20240507 02:08:15.193230 9104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:08:15.193324 9104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:08:15.194196 9104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715047604443657-32269-0/minicluster-data/master-0/data/instance:
uuid: "6348e6c0b66c4bce8c989c83a3b773a3"
format_stamp: "Formatted at 2024-05-07 02:08:15 on dist-test-slave-v0bg"
server_key: "66a2986e660ced95dfc8e64230363595"
server_key_iv: "c1cbfd6c2b80b8ae9cb9f48a6c080225"
server_key_version: "encryptionkey@0"
I20240507 02:08:15.194464 9104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715047604443657-32269-0/minicluster-data/master-0/wal/instance:
uuid: "6348e6c0b66c4bce8c989c83a3b773a3"
format_stamp: "Formatted at 2024-05-07 02:08:15 on dist-test-slave-v0bg"
server_key: "66a2986e660ced95dfc8e64230363595"
server_key_iv: "c1cbfd6c2b80b8ae9cb9f48a6c080225"
server_key_version: "encryptionkey@0"
I20240507 02:08:15.195446 9104 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240507 02:08:15.195984 9118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:08:15.196121 9104 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240507 02:08:15.196187 9104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715047604443657-32269-0/minicluster-data/master-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715047604443657-32269-0/minicluster-data/master-0/wal
uuid: "6348e6c0b66c4bce8c989c83a3b773a3"
format_stamp: "Formatted at 2024-05-07 02:08:15 on dist-test-slave-v0bg"
server_key: "66a2986e660ced95dfc8e64230363595"
server_key_iv: "c1cbfd6c2b80b8ae9cb9f48a6c080225"
server_key_version: "encryptionkey@0"
I20240507 02:08:15.196240 9104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715047604443657-32269-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715047604443657-32269-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715047604443657-32269-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:08:15.212064 9104 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:08:15.212242 9104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:08:15.212332 9104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:08:15.215317 9104 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.126:37637
I20240507 02:08:15.215338 9170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.126:37637 every 8 connection(s)
I20240507 02:08:15.215579 9104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
I20240507 02:08:15.216063 9171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:08:15.218186 9171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6348e6c0b66c4bce8c989c83a3b773a3: Bootstrap starting.
I20240507 02:08:15.218693 9171 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6348e6c0b66c4bce8c989c83a3b773a3: Neither blocks nor log segments found. Creating new log.
I20240507 02:08:15.218911 9171 log.cc:824] T 00000000000000000000000000000000 P 6348e6c0b66c4bce8c989c83a3b773a3: Log is configured to *not* fsync() on all Append() calls
I20240507 02:08:15.219480 9171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6348e6c0b66c4bce8c989c83a3b773a3: No bootstrap required, opened a new log
I20240507 02:08:15.220752 9171 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6348e6c0b66c4bce8c989c83a3b773a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6348e6c0b66c4bce8c989c83a3b773a3" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 37637 } }
I20240507 02:08:15.220853 9171 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6348e6c0b66c4bce8c989c83a3b773a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:08:15.220867 9171 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6348e6c0b66c4bce8c989c83a3b773a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6348e6c0b66c4bce8c989c83a3b773a3, State: Initialized, Role: FOLLOWER
I20240507 02:08:15.220971 9171 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6348e6c0b66c4bce8c989c83a3b773a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6348e6c0b66c4bce8c989c83a3b773a3" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 37637 } }
I20240507 02:08:15.221035 9171 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6348e6c0b66c4bce8c989c83a3b773a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 02:08:15.221055 9171 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6348e6c0b66c4bce8c989c83a3b773a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 02:08:15.221094 9171 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6348e6c0b66c4bce8c989c83a3b773a3 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:08:15.221572 9171 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6348e6c0b66c4bce8c989c83a3b773a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6348e6c0b66c4bce8c989c83a3b773a3" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 37637 } }
I20240507 02:08:15.221648 9171 leader_election.cc:302] T 00000000000000000000000000000000 P 6348e6c0b66c4bce8c989c83a3b773a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6348e6c0b66c4bce8c989c83a3b773a3; no voters:
I20240507 02:08:15.221760 9171 leader_election.cc:288] T 00000000000000000000000000000000 P 6348e6c0b66c4bce8c989c83a3b773a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240507 02:08:15.221812 9174 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6348e6c0b66c4bce8c989c83a3b773a3 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:08:15.221963 9174 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6348e6c0b66c4bce8c989c83a3b773a3 [term 1 LEADER]: Becoming Leader. State: Replica: 6348e6c0b66c4bce8c989c83a3b773a3, State: Running, Role: LEADER
I20240507 02:08:15.221962 9171 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6348e6c0b66c4bce8c989c83a3b773a3 [sys.catalog]: configured and running, proceeding with master startup.
I20240507 02:08:15.222096 9174 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6348e6c0b66c4bce8c989c83a3b773a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6348e6c0b66c4bce8c989c83a3b773a3" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 37637 } }
I20240507 02:08:15.222397 9175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6348e6c0b66c4bce8c989c83a3b773a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6348e6c0b66c4bce8c989c83a3b773a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6348e6c0b66c4bce8c989c83a3b773a3" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 37637 } } }
I20240507 02:08:15.222425 9176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6348e6c0b66c4bce8c989c83a3b773a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6348e6c0b66c4bce8c989c83a3b773a3. Latest consensus state: current_term: 1 leader_uuid: "6348e6c0b66c4bce8c989c83a3b773a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6348e6c0b66c4bce8c989c83a3b773a3" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 37637 } } }
I20240507 02:08:15.222502 9175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6348e6c0b66c4bce8c989c83a3b773a3 [sys.catalog]: This master's current role is: LEADER
I20240507 02:08:15.222503 9176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6348e6c0b66c4bce8c989c83a3b773a3 [sys.catalog]: This master's current role is: LEADER
I20240507 02:08:15.222800 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 9104
I20240507 02:08:15.222851 9182 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 02:08:15.222891 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715047604443657-32269-0/minicluster-data/master-0/wal/instance
I20240507 02:08:15.223047 32269 external_mini_cluster.cc:1428] Setting key 4c88b2444c26c7bff5e2cc681a1c1fbf
I20240507 02:08:15.224316 9182 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240507 02:08:15.225641 9182 catalog_manager.cc:1347] Generated new cluster ID: 6782540fe0314b69b522a6a801c3014b
I20240507 02:08:15.225688 9182 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240507 02:08:15.227756 9190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:15.223487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33076 (local address 127.31.131.126:37637)
0507 02:08:15.223624 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:15.223629 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:15.223652 (+ 23us) server_negotiation.cc:407] Connection header received
0507 02:08:15.223792 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:15.223797 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:15.223856 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:15.223981 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:08:15.224532 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:15.225440 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:08:15.226185 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:15.226404 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:15.227124 (+ 720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:08:15.227144 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:08:15.227145 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0507 02:08:15.227321 (+ 176us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:08:15.227348 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0507 02:08:15.227355 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:08:15.227397 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0507 02:08:15.227544 (+ 147us) server_negotiation.cc:299] Negotiation successful
0507 02:08:15.227593 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74,"thread_start_us":59,"threads_started":1}
I20240507 02:08:15.232518 9182 catalog_manager.cc:1370] Generated new certificate authority record
I20240507 02:08:15.232941 9182 catalog_manager.cc:1504] Loading token signing keys...
I20240507 02:08:15.238765 9182 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 6348e6c0b66c4bce8c989c83a3b773a3: Generated new TSK 0
I20240507 02:08:15.238891 9182 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240507 02:08:15.241497 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.65:0
--local_ip_for_outbound_sockets=127.31.131.65
--webserver_interface=127.31.131.65
--webserver_port=0
--tserver_master_addrs=127.31.131.126:37637
--builtin_ntp_servers=127.31.131.84:41833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20240507 02:08:15.313495 9195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:08:15.313633 9195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:08:15.313656 9195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:08:15.314936 9195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:08:15.314971 9195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:08:15.315001 9195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.65
I20240507 02:08:15.316301 9195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:41833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.131.65
--webserver_port=0
--tserver_master_addrs=127.31.131.126:37637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.65
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:08:15.316484 9195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:08:15.316673 9195 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:08:15.320605 9201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:08:15.320672 9200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:08:15.320727 9203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:08:15.320924 9195 server_base.cc:1034] running on GCE node
I20240507 02:08:15.321069 9195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:08:15.321236 9195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:08:15.322335 9195 hybrid_clock.cc:648] HybridClock initialized: now 1715047695322319 us; error 28 us; skew 500 ppm
I20240507 02:08:15.323501 9195 webserver.cc:460] Webserver started at http://127.31.131.65:33417/ using document root <none> and password file <none>
I20240507 02:08:15.323671 9195 fs_manager.cc:359] Metadata directory not provided
I20240507 02:08:15.323707 9195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:08:15.323794 9195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:08:15.324677 9195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-0/data/instance:
uuid: "7ed079c0eca047bca66d4f60901c1dc6"
format_stamp: "Formatted at 2024-05-07 02:08:15 on dist-test-slave-v0bg"
server_key: "435754e5e73fa9bf55f1559ea305e3d2"
server_key_iv: "eb875d232aee8205dda594bdbb5473e9"
server_key_version: "encryptionkey@0"
I20240507 02:08:15.324935 9195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-0/wal/instance:
uuid: "7ed079c0eca047bca66d4f60901c1dc6"
format_stamp: "Formatted at 2024-05-07 02:08:15 on dist-test-slave-v0bg"
server_key: "435754e5e73fa9bf55f1559ea305e3d2"
server_key_iv: "eb875d232aee8205dda594bdbb5473e9"
server_key_version: "encryptionkey@0"
I20240507 02:08:15.325939 9195 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240507 02:08:15.326524 9209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:08:15.326690 9195 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240507 02:08:15.326756 9195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-0/wal
uuid: "7ed079c0eca047bca66d4f60901c1dc6"
format_stamp: "Formatted at 2024-05-07 02:08:15 on dist-test-slave-v0bg"
server_key: "435754e5e73fa9bf55f1559ea305e3d2"
server_key_iv: "eb875d232aee8205dda594bdbb5473e9"
server_key_version: "encryptionkey@0"
I20240507 02:08:15.326810 9195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:08:15.348819 9195 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:08:15.349011 9195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:08:15.349100 9195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:08:15.349264 9195 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:08:15.349503 9195 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:08:15.349532 9195 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:08:15.349557 9195 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:08:15.349581 9195 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:08:15.354702 9195 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.65:35061
I20240507 02:08:15.354745 9323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.65:35061 every 8 connection(s)
I20240507 02:08:15.355001 9195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
I20240507 02:08:15.359416 9190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:15.355604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:42657 (local address 127.31.131.126:37637)
0507 02:08:15.355652 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:15.355654 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:15.355833 (+ 179us) server_negotiation.cc:407] Connection header received
0507 02:08:15.356373 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:15.356379 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:15.356388 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:15.356433 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:08:15.356956 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:15.357409 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:08:15.358135 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:15.358240 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:15.359196 (+ 956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:08:15.359199 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:08:15.359200 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0507 02:08:15.359203 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:08:15.359217 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0507 02:08:15.359220 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:08:15.359258 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0507 02:08:15.359334 (+ 76us) server_negotiation.cc:299] Negotiation successful
0507 02:08:15.359357 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20240507 02:08:15.359613 9326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:15.355578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:37637 (local address 127.31.131.65:42657)
0507 02:08:15.355736 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0507 02:08:15.355761 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0507 02:08:15.356279 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:08:15.356484 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:08:15.356488 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:08:15.356504 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:08:15.356886 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:15.356890 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:15.357482 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:08:15.357483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:08:15.358066 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:15.358068 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:15.358179 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:15.359076 (+ 897us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0507 02:08:15.359077 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:08:15.359097 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0507 02:08:15.359115 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:08:15.359272 (+ 157us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:08:15.359274 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:08:15.359276 (+ 2us) client_negotiation.cc:769] Sending connection context
0507 02:08:15.359337 (+ 61us) client_negotiation.cc:240] Negotiation successful
0507 02:08:15.359402 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":68,"threads_started":1}
I20240507 02:08:15.360045 9324 heartbeater.cc:347] Connected to a master server at 127.31.131.126:37637
I20240507 02:08:15.360121 9324 heartbeater.cc:464] Registering TS with master...
I20240507 02:08:15.360296 9324 heartbeater.cc:511] Master 127.31.131.126:37637 requested a full tablet report, sending...
I20240507 02:08:15.360677 9135 ts_manager.cc:194] Registered new tserver with Master: 7ed079c0eca047bca66d4f60901c1dc6 (127.31.131.65:35061)
I20240507 02:08:15.361243 9135 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.131.65:42657
I20240507 02:08:15.365089 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 9195
I20240507 02:08:15.365188 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-0/wal/instance
I20240507 02:08:15.365325 32269 external_mini_cluster.cc:1428] Setting key 697d7ecfcd1583957fdb7fb4892fc9f8
I20240507 02:08:15.366075 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.66:0
--local_ip_for_outbound_sockets=127.31.131.66
--webserver_interface=127.31.131.66
--webserver_port=0
--tserver_master_addrs=127.31.131.126:37637
--builtin_ntp_servers=127.31.131.84:41833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20240507 02:08:15.437906 9327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:08:15.438037 9327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:08:15.438053 9327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:08:15.439270 9327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:08:15.439302 9327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:08:15.439322 9327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.66
I20240507 02:08:15.440613 9327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:41833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.131.66
--webserver_port=0
--tserver_master_addrs=127.31.131.126:37637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.66
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:08:15.440742 9327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:08:15.440907 9327 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:08:15.444213 9333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:08:15.444214 9332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:08:15.444386 9327 server_base.cc:1034] running on GCE node
W20240507 02:08:15.444401 9335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:08:15.444650 9327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:08:15.444810 9327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:08:15.445932 9327 hybrid_clock.cc:648] HybridClock initialized: now 1715047695445926 us; error 25 us; skew 500 ppm
I20240507 02:08:15.446946 9327 webserver.cc:460] Webserver started at http://127.31.131.66:35531/ using document root <none> and password file <none>
I20240507 02:08:15.447089 9327 fs_manager.cc:359] Metadata directory not provided
I20240507 02:08:15.447124 9327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:08:15.447213 9327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:08:15.448040 9327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-1/data/instance:
uuid: "35e83d11878d4b8a92ca35cc8cd82f4e"
format_stamp: "Formatted at 2024-05-07 02:08:15 on dist-test-slave-v0bg"
server_key: "a21d9bdb72746bb5dfcd6b5119b17897"
server_key_iv: "051dba772ca8520403803991d572ef83"
server_key_version: "encryptionkey@0"
I20240507 02:08:15.448287 9327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-1/wal/instance:
uuid: "35e83d11878d4b8a92ca35cc8cd82f4e"
format_stamp: "Formatted at 2024-05-07 02:08:15 on dist-test-slave-v0bg"
server_key: "a21d9bdb72746bb5dfcd6b5119b17897"
server_key_iv: "051dba772ca8520403803991d572ef83"
server_key_version: "encryptionkey@0"
I20240507 02:08:15.449224 9327 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240507 02:08:15.449754 9341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:08:15.449883 9327 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240507 02:08:15.449942 9327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-1/wal
uuid: "35e83d11878d4b8a92ca35cc8cd82f4e"
format_stamp: "Formatted at 2024-05-07 02:08:15 on dist-test-slave-v0bg"
server_key: "a21d9bdb72746bb5dfcd6b5119b17897"
server_key_iv: "051dba772ca8520403803991d572ef83"
server_key_version: "encryptionkey@0"
I20240507 02:08:15.449997 9327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:08:15.467397 9327 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:08:15.467581 9327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:08:15.467654 9327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:08:15.467780 9327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:08:15.468000 9327 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:08:15.468029 9327 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:08:15.468052 9327 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:08:15.468067 9327 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:08:15.472575 9327 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.66:34973
I20240507 02:08:15.472606 9454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.66:34973 every 8 connection(s)
I20240507 02:08:15.472818 9327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
I20240507 02:08:15.476291 9190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:15.473361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.66:52775 (local address 127.31.131.126:37637)
0507 02:08:15.473408 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:15.473410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:15.473584 (+ 174us) server_negotiation.cc:407] Connection header received
0507 02:08:15.473945 (+ 361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:15.473946 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:15.473954 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:15.474000 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:08:15.474375 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:15.474865 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:08:15.475325 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:15.475424 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:15.476091 (+ 667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:08:15.476094 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:08:15.476094 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0507 02:08:15.476098 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:08:15.476111 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0507 02:08:15.476114 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:08:15.476151 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0507 02:08:15.476213 (+ 62us) server_negotiation.cc:299] Negotiation successful
0507 02:08:15.476240 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20240507 02:08:15.476367 9457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:15.473331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:37637 (local address 127.31.131.66:52775)
0507 02:08:15.473501 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0507 02:08:15.473521 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0507 02:08:15.473875 (+ 354us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:08:15.474042 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:08:15.474045 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:08:15.474052 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:08:15.474316 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:15.474319 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:15.474931 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:08:15.474931 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:08:15.475273 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:15.475274 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:15.475346 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:15.475993 (+ 647us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0507 02:08:15.475994 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:08:15.476007 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0507 02:08:15.476022 (+ 15us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:08:15.476159 (+ 137us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:08:15.476160 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:08:15.476162 (+ 2us) client_negotiation.cc:769] Sending connection context
0507 02:08:15.476206 (+ 44us) client_negotiation.cc:240] Negotiation successful
0507 02:08:15.476255 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":77,"threads_started":1}
I20240507 02:08:15.476770 9455 heartbeater.cc:347] Connected to a master server at 127.31.131.126:37637
I20240507 02:08:15.476838 9455 heartbeater.cc:464] Registering TS with master...
I20240507 02:08:15.477002 9455 heartbeater.cc:511] Master 127.31.131.126:37637 requested a full tablet report, sending...
I20240507 02:08:15.477303 9135 ts_manager.cc:194] Registered new tserver with Master: 35e83d11878d4b8a92ca35cc8cd82f4e (127.31.131.66:34973)
I20240507 02:08:15.477641 9135 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.131.66:52775
I20240507 02:08:15.479308 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 9327
I20240507 02:08:15.479399 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-1/wal/instance
I20240507 02:08:15.479542 32269 external_mini_cluster.cc:1428] Setting key 8837b1f1585e419ff5e7417b339b52bd
I20240507 02:08:15.480274 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.67:0
--local_ip_for_outbound_sockets=127.31.131.67
--webserver_interface=127.31.131.67
--webserver_port=0
--tserver_master_addrs=127.31.131.126:37637
--builtin_ntp_servers=127.31.131.84:41833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20240507 02:08:15.553674 9458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:08:15.553803 9458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:08:15.553820 9458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:08:15.555087 9458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:08:15.555122 9458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:08:15.555145 9458 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.67
I20240507 02:08:15.556453 9458 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:41833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.131.67
--webserver_port=0
--tserver_master_addrs=127.31.131.126:37637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.67
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:08:15.556622 9458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:08:15.556807 9458 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:08:15.560115 9466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:08:15.560115 9464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:08:15.560271 9463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:08:15.560364 9458 server_base.cc:1034] running on GCE node
I20240507 02:08:15.560510 9458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:08:15.560684 9458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:08:15.561800 9458 hybrid_clock.cc:648] HybridClock initialized: now 1715047695561784 us; error 26 us; skew 500 ppm
I20240507 02:08:15.562783 9458 webserver.cc:460] Webserver started at http://127.31.131.67:46823/ using document root <none> and password file <none>
I20240507 02:08:15.562933 9458 fs_manager.cc:359] Metadata directory not provided
I20240507 02:08:15.562970 9458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:08:15.563060 9458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:08:15.563880 9458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-2/data/instance:
uuid: "c1cb76677a9c47e4a63db65a05da69cc"
format_stamp: "Formatted at 2024-05-07 02:08:15 on dist-test-slave-v0bg"
server_key: "f1e0405d1f7daff409f4ba76bd93549e"
server_key_iv: "fb32a765de896519855a423228022217"
server_key_version: "encryptionkey@0"
I20240507 02:08:15.564131 9458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-2/wal/instance:
uuid: "c1cb76677a9c47e4a63db65a05da69cc"
format_stamp: "Formatted at 2024-05-07 02:08:15 on dist-test-slave-v0bg"
server_key: "f1e0405d1f7daff409f4ba76bd93549e"
server_key_iv: "fb32a765de896519855a423228022217"
server_key_version: "encryptionkey@0"
I20240507 02:08:15.565094 9458 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240507 02:08:15.565629 9472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:08:15.565763 9458 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240507 02:08:15.565825 9458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-2/wal
uuid: "c1cb76677a9c47e4a63db65a05da69cc"
format_stamp: "Formatted at 2024-05-07 02:08:15 on dist-test-slave-v0bg"
server_key: "f1e0405d1f7daff409f4ba76bd93549e"
server_key_iv: "fb32a765de896519855a423228022217"
server_key_version: "encryptionkey@0"
I20240507 02:08:15.565876 9458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:08:15.591635 9458 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:08:15.591787 9458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:08:15.591873 9458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:08:15.592013 9458 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:08:15.592236 9458 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:08:15.592265 9458 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:08:15.592289 9458 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:08:15.592305 9458 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:08:15.599586 9458 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.67:37663
I20240507 02:08:15.599676 9585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.67:37663 every 8 connection(s)
I20240507 02:08:15.599895 9458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
I20240507 02:08:15.603631 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 9458
I20240507 02:08:15.603732 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-2/wal/instance
I20240507 02:08:15.603910 32269 external_mini_cluster.cc:1428] Setting key dbca6a77355785de23de905c97b97eb4
I20240507 02:08:15.604212 9190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:15.600506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.67:41471 (local address 127.31.131.126:37637)
0507 02:08:15.600555 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:15.600558 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:15.600763 (+ 205us) server_negotiation.cc:407] Connection header received
0507 02:08:15.601246 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:15.601247 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:15.601264 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:15.601312 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:08:15.601783 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:15.602442 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:08:15.603025 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:15.603198 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:15.603955 (+ 757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:08:15.603960 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:08:15.603961 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0507 02:08:15.603966 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:08:15.603984 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0507 02:08:15.603988 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:08:15.604037 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0507 02:08:15.604113 (+ 76us) server_negotiation.cc:299] Negotiation successful
0507 02:08:15.604151 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20240507 02:08:15.604362 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:15.600486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:37637 (local address 127.31.131.67:41471)
0507 02:08:15.600656 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0507 02:08:15.600679 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0507 02:08:15.601149 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:08:15.601357 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:08:15.601361 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:08:15.601370 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:08:15.601704 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:15.601709 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:15.602523 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:08:15.602524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:08:15.602950 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:15.602952 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:15.603052 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:15.603831 (+ 779us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0507 02:08:15.603832 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:08:15.603854 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0507 02:08:15.603873 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:08:15.604050 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:08:15.604054 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:08:15.604056 (+ 2us) client_negotiation.cc:769] Sending connection context
0507 02:08:15.604105 (+ 49us) client_negotiation.cc:240] Negotiation successful
0507 02:08:15.604167 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":78,"threads_started":1}
I20240507 02:08:15.604753 9586 heartbeater.cc:347] Connected to a master server at 127.31.131.126:37637
I20240507 02:08:15.604833 9586 heartbeater.cc:464] Registering TS with master...
I20240507 02:08:15.604985 9586 heartbeater.cc:511] Master 127.31.131.126:37637 requested a full tablet report, sending...
I20240507 02:08:15.605273 9135 ts_manager.cc:194] Registered new tserver with Master: c1cb76677a9c47e4a63db65a05da69cc (127.31.131.67:37663)
I20240507 02:08:15.605648 9135 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.131.67:41471
I20240507 02:08:15.615480 32269 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20240507 02:08:15.616974 9190 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:08:15.616573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33090 (local address 127.31.131.126:37637)
0507 02:08:15.616617 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:15.616619 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:15.616716 (+ 97us) server_negotiation.cc:407] Connection header received
0507 02:08:15.616772 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:15.616773 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:15.616782 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:15.616824 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:08:15.616938 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33090: BlockingRecv error: recv got EOF from 127.0.0.1:33090 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
I20240507 02:08:15.617074 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 9195
I20240507 02:08:15.621030 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 9327
I20240507 02:08:15.624701 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 9458
I20240507 02:08:15.628077 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 9104
2024-05-07T02:08:15Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (540 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2024-05-07T02:08:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-07T02:08:15Z Disabled control of system clock
I20240507 02:08:15.650215 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715047604443657-32269-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715047604443657-32269-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715047604443657-32269-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:36577
--webserver_interface=127.31.131.126
--webserver_port=0
--builtin_ntp_servers=127.31.131.84:33643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.131.126:36577
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20240507 02:08:15.722054 9597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:08:15.722199 9597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:08:15.722225 9597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:08:15.723444 9597 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240507 02:08:15.723484 9597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:08:15.723500 9597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:08:15.723515 9597 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240507 02:08:15.723528 9597 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240507 02:08:15.724870 9597 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:33643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715047604443657-32269-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715047604443657-32269-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.131.126:36577
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:36577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.131.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9597
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715047604443657-32269-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:08:15.725059 9597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:08:15.725252 9597 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:08:15.728446 9602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:08:15.728442 9603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:08:15.728572 9597 server_base.cc:1034] running on GCE node
W20240507 02:08:15.728442 9605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:08:15.728804 9597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:08:15.728991 9597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:08:15.730108 9597 hybrid_clock.cc:648] HybridClock initialized: now 1715047695730100 us; error 22 us; skew 500 ppm
I20240507 02:08:15.731020 9597 webserver.cc:460] Webserver started at http://127.31.131.126:32981/ using document root <none> and password file <none>
I20240507 02:08:15.731168 9597 fs_manager.cc:359] Metadata directory not provided
I20240507 02:08:15.731204 9597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:08:15.731307 9597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:08:15.732163 9597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715047604443657-32269-0/minicluster-data/master-0/data/instance:
uuid: "66fee7873c75426fb525329ad6e494cc"
format_stamp: "Formatted at 2024-05-07 02:08:15 on dist-test-slave-v0bg"
server_key: "ce4ad001d8c6c1d739c1217188ffa458"
server_key_iv: "a7ff75b44956776de018f7371a832755"
server_key_version: "encryptionkey@0"
I20240507 02:08:15.732422 9597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715047604443657-32269-0/minicluster-data/master-0/wal/instance:
uuid: "66fee7873c75426fb525329ad6e494cc"
format_stamp: "Formatted at 2024-05-07 02:08:15 on dist-test-slave-v0bg"
server_key: "ce4ad001d8c6c1d739c1217188ffa458"
server_key_iv: "a7ff75b44956776de018f7371a832755"
server_key_version: "encryptionkey@0"
I20240507 02:08:15.733373 9597 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240507 02:08:15.733891 9611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:08:15.734040 9597 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240507 02:08:15.734117 9597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715047604443657-32269-0/minicluster-data/master-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715047604443657-32269-0/minicluster-data/master-0/wal
uuid: "66fee7873c75426fb525329ad6e494cc"
format_stamp: "Formatted at 2024-05-07 02:08:15 on dist-test-slave-v0bg"
server_key: "ce4ad001d8c6c1d739c1217188ffa458"
server_key_iv: "a7ff75b44956776de018f7371a832755"
server_key_version: "encryptionkey@0"
I20240507 02:08:15.734169 9597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715047604443657-32269-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715047604443657-32269-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715047604443657-32269-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:08:15.737222 9597 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:08:15.737444 9597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:08:15.737532 9597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:08:15.740721 9597 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.126:36577
I20240507 02:08:15.740746 9663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.126:36577 every 8 connection(s)
I20240507 02:08:15.740967 9597 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
I20240507 02:08:15.741413 9664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:08:15.742821 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 9597
I20240507 02:08:15.742920 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715047604443657-32269-0/minicluster-data/master-0/wal/instance
I20240507 02:08:15.743074 32269 external_mini_cluster.cc:1428] Setting key e460fa2bf2ecebfd13eb0b5ba2d58e72
I20240507 02:08:15.743606 9664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66fee7873c75426fb525329ad6e494cc: Bootstrap starting.
I20240507 02:08:15.744055 9664 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66fee7873c75426fb525329ad6e494cc: Neither blocks nor log segments found. Creating new log.
I20240507 02:08:15.744302 9664 log.cc:824] T 00000000000000000000000000000000 P 66fee7873c75426fb525329ad6e494cc: Log is configured to *not* fsync() on all Append() calls
I20240507 02:08:15.744904 9664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66fee7873c75426fb525329ad6e494cc: No bootstrap required, opened a new log
I20240507 02:08:15.745108 9667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:15.743473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37740 (local address 127.31.131.126:36577)
0507 02:08:15.743626 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:15.743631 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:15.743646 (+ 15us) server_negotiation.cc:407] Connection header received
0507 02:08:15.744057 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:15.744062 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:15.744092 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:15.744206 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:08:15.744789 (+ 583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:08:15.744792 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:08:15.744792 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0507 02:08:15.744796 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:08:15.744824 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0507 02:08:15.744830 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:08:15.744913 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0507 02:08:15.744945 (+ 32us) server_negotiation.cc:299] Negotiation successful
0507 02:08:15.744983 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":64,"threads_started":1}
I20240507 02:08:15.746104 9664 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66fee7873c75426fb525329ad6e494cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66fee7873c75426fb525329ad6e494cc" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 36577 } }
I20240507 02:08:15.746203 9664 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66fee7873c75426fb525329ad6e494cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:08:15.746219 9664 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66fee7873c75426fb525329ad6e494cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66fee7873c75426fb525329ad6e494cc, State: Initialized, Role: FOLLOWER
I20240507 02:08:15.746294 9664 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66fee7873c75426fb525329ad6e494cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66fee7873c75426fb525329ad6e494cc" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 36577 } }
I20240507 02:08:15.746352 9664 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66fee7873c75426fb525329ad6e494cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 02:08:15.746374 9664 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66fee7873c75426fb525329ad6e494cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 02:08:15.746423 9664 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66fee7873c75426fb525329ad6e494cc [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:08:15.746963 9664 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66fee7873c75426fb525329ad6e494cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66fee7873c75426fb525329ad6e494cc" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 36577 } }
I20240507 02:08:15.747058 9664 leader_election.cc:302] T 00000000000000000000000000000000 P 66fee7873c75426fb525329ad6e494cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66fee7873c75426fb525329ad6e494cc; no voters:
I20240507 02:08:15.747190 9664 leader_election.cc:288] T 00000000000000000000000000000000 P 66fee7873c75426fb525329ad6e494cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20240507 02:08:15.747227 9669 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66fee7873c75426fb525329ad6e494cc [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:08:15.747347 9669 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66fee7873c75426fb525329ad6e494cc [term 1 LEADER]: Becoming Leader. State: Replica: 66fee7873c75426fb525329ad6e494cc, State: Running, Role: LEADER
I20240507 02:08:15.747368 9664 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66fee7873c75426fb525329ad6e494cc [sys.catalog]: configured and running, proceeding with master startup.
I20240507 02:08:15.747422 9669 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66fee7873c75426fb525329ad6e494cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66fee7873c75426fb525329ad6e494cc" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 36577 } }
I20240507 02:08:15.747721 9670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66fee7873c75426fb525329ad6e494cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66fee7873c75426fb525329ad6e494cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66fee7873c75426fb525329ad6e494cc" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 36577 } } }
I20240507 02:08:15.747758 9671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66fee7873c75426fb525329ad6e494cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66fee7873c75426fb525329ad6e494cc. Latest consensus state: current_term: 1 leader_uuid: "66fee7873c75426fb525329ad6e494cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66fee7873c75426fb525329ad6e494cc" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 36577 } } }
I20240507 02:08:15.747859 9670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66fee7873c75426fb525329ad6e494cc [sys.catalog]: This master's current role is: LEADER
I20240507 02:08:15.747947 9671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66fee7873c75426fb525329ad6e494cc [sys.catalog]: This master's current role is: LEADER
I20240507 02:08:15.748203 9676 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 02:08:15.748585 9676 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240507 02:08:15.749718 9676 catalog_manager.cc:1347] Generated new cluster ID: 1daa56266ad84dcf80bfa62fd9986a6b
I20240507 02:08:15.749758 9676 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240507 02:08:15.754901 9676 catalog_manager.cc:1370] Generated new certificate authority record
I20240507 02:08:15.755004 9676 catalog_manager.cc:1504] Loading token signing keys...
I20240507 02:08:15.760324 9676 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 66fee7873c75426fb525329ad6e494cc: Generated new TSK 0
I20240507 02:08:15.760452 9676 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240507 02:08:15.765254 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.65:0
--local_ip_for_outbound_sockets=127.31.131.65
--webserver_interface=127.31.131.65
--webserver_port=0
--tserver_master_addrs=127.31.131.126:36577
--builtin_ntp_servers=127.31.131.84:33643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20240507 02:08:15.853580 9688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:08:15.853719 9688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:08:15.853739 9688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:08:15.854975 9688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:08:15.855011 9688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:08:15.855033 9688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.65
I20240507 02:08:15.856338 9688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:33643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.131.65
--webserver_port=0
--tserver_master_addrs=127.31.131.126:36577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.65
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:08:15.856504 9688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:08:15.856690 9688 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:08:15.860597 9696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:08:15.860620 9693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:08:15.860603 9694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:08:15.860875 9688 server_base.cc:1034] running on GCE node
I20240507 02:08:15.861034 9688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:08:15.861238 9688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:08:15.862394 9688 hybrid_clock.cc:648] HybridClock initialized: now 1715047695862377 us; error 29 us; skew 500 ppm
I20240507 02:08:15.863665 9688 webserver.cc:460] Webserver started at http://127.31.131.65:42215/ using document root <none> and password file <none>
I20240507 02:08:15.863863 9688 fs_manager.cc:359] Metadata directory not provided
I20240507 02:08:15.863919 9688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:08:15.864023 9688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:08:15.865170 9688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-0/data/instance:
uuid: "dd9b0c7f509d474a84ea3e48529055d1"
format_stamp: "Formatted at 2024-05-07 02:08:15 on dist-test-slave-v0bg"
server_key: "9a4bb7b5b579958b28dcf9dc336cc0ba"
server_key_iv: "e201ff26197030a797a289e6384d2992"
server_key_version: "encryptionkey@0"
I20240507 02:08:15.865517 9688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-0/wal/instance:
uuid: "dd9b0c7f509d474a84ea3e48529055d1"
format_stamp: "Formatted at 2024-05-07 02:08:15 on dist-test-slave-v0bg"
server_key: "9a4bb7b5b579958b28dcf9dc336cc0ba"
server_key_iv: "e201ff26197030a797a289e6384d2992"
server_key_version: "encryptionkey@0"
I20240507 02:08:15.866881 9688 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240507 02:08:15.867594 9702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:08:15.867744 9688 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240507 02:08:15.867825 9688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-0/wal
uuid: "dd9b0c7f509d474a84ea3e48529055d1"
format_stamp: "Formatted at 2024-05-07 02:08:15 on dist-test-slave-v0bg"
server_key: "9a4bb7b5b579958b28dcf9dc336cc0ba"
server_key_iv: "e201ff26197030a797a289e6384d2992"
server_key_version: "encryptionkey@0"
I20240507 02:08:15.867884 9688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:08:15.872566 9688 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:08:15.872776 9688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:08:15.872874 9688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:08:15.873056 9688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:08:15.873337 9688 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:08:15.873380 9688 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:08:15.873418 9688 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:08:15.873445 9688 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:08:15.879561 9688 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.65:39333
I20240507 02:08:15.879595 9815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.65:39333 every 8 connection(s)
I20240507 02:08:15.879858 9688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
I20240507 02:08:15.882030 9667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:15.880426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:43367 (local address 127.31.131.126:36577)
0507 02:08:15.880482 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:15.880484 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:15.880685 (+ 201us) server_negotiation.cc:407] Connection header received
0507 02:08:15.881186 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:15.881187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:15.881192 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:15.881226 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:08:15.881836 (+ 610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:08:15.881837 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:08:15.881837 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0507 02:08:15.881840 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:08:15.881852 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0507 02:08:15.881856 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:08:15.881904 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0507 02:08:15.881972 (+ 68us) server_negotiation.cc:299] Negotiation successful
0507 02:08:15.881982 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20240507 02:08:15.882189 9818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:15.880368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:36577 (local address 127.31.131.65:43367)
0507 02:08:15.880578 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0507 02:08:15.880604 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0507 02:08:15.881094 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:08:15.881285 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:08:15.881289 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:08:15.881298 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:08:15.881718 (+ 420us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0507 02:08:15.881719 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:08:15.881739 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0507 02:08:15.881763 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:08:15.881907 (+ 144us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:08:15.881909 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:08:15.881912 (+ 3us) client_negotiation.cc:769] Sending connection context
0507 02:08:15.881993 (+ 81us) client_negotiation.cc:240] Negotiation successful
0507 02:08:15.882035 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":75,"threads_started":1}
I20240507 02:08:15.882908 9816 heartbeater.cc:347] Connected to a master server at 127.31.131.126:36577
I20240507 02:08:15.882993 9816 heartbeater.cc:464] Registering TS with master...
I20240507 02:08:15.883101 9816 heartbeater.cc:511] Master 127.31.131.126:36577 requested a full tablet report, sending...
I20240507 02:08:15.883579 9628 ts_manager.cc:194] Registered new tserver with Master: dd9b0c7f509d474a84ea3e48529055d1 (127.31.131.65:39333)
I20240507 02:08:15.888983 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 9688
I20240507 02:08:15.889075 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-0/wal/instance
I20240507 02:08:15.889209 32269 external_mini_cluster.cc:1428] Setting key b0619d9f9f53bfa102f6d3f61946ea90
I20240507 02:08:15.889999 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.66:0
--local_ip_for_outbound_sockets=127.31.131.66
--webserver_interface=127.31.131.66
--webserver_port=0
--tserver_master_addrs=127.31.131.126:36577
--builtin_ntp_servers=127.31.131.84:33643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20240507 02:08:15.963058 9819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:08:15.963193 9819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:08:15.963212 9819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:08:15.964414 9819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:08:15.964452 9819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:08:15.964475 9819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.66
I20240507 02:08:15.965795 9819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:33643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.131.66
--webserver_port=0
--tserver_master_addrs=127.31.131.126:36577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.66
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:08:15.965961 9819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:08:15.966152 9819 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240507 02:08:15.969713 9819 server_base.cc:1034] running on GCE node
W20240507 02:08:15.969748 9825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:08:15.969851 9827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:08:15.969870 9824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:08:15.970052 9819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:08:15.970242 9819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:08:15.971359 9819 hybrid_clock.cc:648] HybridClock initialized: now 1715047695971346 us; error 27 us; skew 500 ppm
I20240507 02:08:15.972347 9819 webserver.cc:460] Webserver started at http://127.31.131.66:37049/ using document root <none> and password file <none>
I20240507 02:08:15.972502 9819 fs_manager.cc:359] Metadata directory not provided
I20240507 02:08:15.972541 9819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:08:15.972635 9819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:08:15.973454 9819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-1/data/instance:
uuid: "58502c4258004a9a91e4b798dfaf67eb"
format_stamp: "Formatted at 2024-05-07 02:08:15 on dist-test-slave-v0bg"
server_key: "5eb2024effbe3f948216a33ae95ec61f"
server_key_iv: "e2eca8ef265e7f930be6d238c5fb5645"
server_key_version: "encryptionkey@0"
I20240507 02:08:15.973721 9819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-1/wal/instance:
uuid: "58502c4258004a9a91e4b798dfaf67eb"
format_stamp: "Formatted at 2024-05-07 02:08:15 on dist-test-slave-v0bg"
server_key: "5eb2024effbe3f948216a33ae95ec61f"
server_key_iv: "e2eca8ef265e7f930be6d238c5fb5645"
server_key_version: "encryptionkey@0"
I20240507 02:08:15.974732 9819 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240507 02:08:15.975260 9833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:08:15.975414 9819 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240507 02:08:15.975478 9819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-1/wal
uuid: "58502c4258004a9a91e4b798dfaf67eb"
format_stamp: "Formatted at 2024-05-07 02:08:15 on dist-test-slave-v0bg"
server_key: "5eb2024effbe3f948216a33ae95ec61f"
server_key_iv: "e2eca8ef265e7f930be6d238c5fb5645"
server_key_version: "encryptionkey@0"
I20240507 02:08:15.975529 9819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:08:15.978610 9819 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:08:15.978758 9819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:08:15.978843 9819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:08:15.978981 9819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:08:15.979216 9819 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:08:15.979245 9819 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:08:15.979269 9819 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:08:15.979297 9819 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:08:15.983700 9819 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.66:46337
I20240507 02:08:15.983748 9946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.66:46337 every 8 connection(s)
I20240507 02:08:15.983947 9819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
I20240507 02:08:15.985860 9667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:15.984489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.66:53895 (local address 127.31.131.126:36577)
0507 02:08:15.984550 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:15.984553 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:15.984682 (+ 129us) server_negotiation.cc:407] Connection header received
0507 02:08:15.985053 (+ 371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:15.985054 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:15.985060 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:15.985115 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:08:15.985636 (+ 521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:08:15.985638 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:08:15.985638 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0507 02:08:15.985642 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:08:15.985656 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0507 02:08:15.985659 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:08:15.985709 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0507 02:08:15.985776 (+ 67us) server_negotiation.cc:299] Negotiation successful
0507 02:08:15.985791 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20240507 02:08:15.985971 9949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:15.984443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:36577 (local address 127.31.131.66:53895)
0507 02:08:15.984604 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0507 02:08:15.984623 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0507 02:08:15.984963 (+ 340us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:08:15.985162 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:08:15.985165 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:08:15.985172 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:08:15.985537 (+ 365us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0507 02:08:15.985537 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:08:15.985550 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0507 02:08:15.985566 (+ 16us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:08:15.985712 (+ 146us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:08:15.985715 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:08:15.985718 (+ 3us) client_negotiation.cc:769] Sending connection context
0507 02:08:15.985795 (+ 77us) client_negotiation.cc:240] Negotiation successful
0507 02:08:15.985835 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":78,"threads_started":1}
I20240507 02:08:15.986522 9947 heartbeater.cc:347] Connected to a master server at 127.31.131.126:36577
I20240507 02:08:15.986621 9947 heartbeater.cc:464] Registering TS with master...
I20240507 02:08:15.986743 9947 heartbeater.cc:511] Master 127.31.131.126:36577 requested a full tablet report, sending...
I20240507 02:08:15.987071 9628 ts_manager.cc:194] Registered new tserver with Master: 58502c4258004a9a91e4b798dfaf67eb (127.31.131.66:46337)
I20240507 02:08:15.993317 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 9819
I20240507 02:08:15.993427 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-1/wal/instance
I20240507 02:08:15.993571 32269 external_mini_cluster.cc:1428] Setting key 74982864d59415bea83c8910c374ec35
I20240507 02:08:15.994366 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.67:0
--local_ip_for_outbound_sockets=127.31.131.67
--webserver_interface=127.31.131.67
--webserver_port=0
--tserver_master_addrs=127.31.131.126:36577
--builtin_ntp_servers=127.31.131.84:33643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20240507 02:08:16.066879 9950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:08:16.067018 9950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:08:16.067039 9950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:08:16.068245 9950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:08:16.068287 9950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:08:16.068311 9950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.67
I20240507 02:08:16.069665 9950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:33643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.131.67
--webserver_port=0
--tserver_master_addrs=127.31.131.126:36577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9950
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.67
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:08:16.069860 9950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:08:16.070060 9950 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:08:16.073570 9955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:08:16.073614 9956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:08:16.073690 9958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:08:16.073808 9950 server_base.cc:1034] running on GCE node
I20240507 02:08:16.073971 9950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:08:16.074157 9950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:08:16.075295 9950 hybrid_clock.cc:648] HybridClock initialized: now 1715047696075279 us; error 26 us; skew 500 ppm
I20240507 02:08:16.076318 9950 webserver.cc:460] Webserver started at http://127.31.131.67:45141/ using document root <none> and password file <none>
I20240507 02:08:16.076483 9950 fs_manager.cc:359] Metadata directory not provided
I20240507 02:08:16.076522 9950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:08:16.076618 9950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:08:16.077455 9950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-2/data/instance:
uuid: "5c7b3b793c584ec4a1446f92ea69b09d"
format_stamp: "Formatted at 2024-05-07 02:08:16 on dist-test-slave-v0bg"
server_key: "f60388f088beeaaa618df3f2bd9f384c"
server_key_iv: "4c6e9651699499897691095a7ab22a5a"
server_key_version: "encryptionkey@0"
I20240507 02:08:16.077721 9950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-2/wal/instance:
uuid: "5c7b3b793c584ec4a1446f92ea69b09d"
format_stamp: "Formatted at 2024-05-07 02:08:16 on dist-test-slave-v0bg"
server_key: "f60388f088beeaaa618df3f2bd9f384c"
server_key_iv: "4c6e9651699499897691095a7ab22a5a"
server_key_version: "encryptionkey@0"
I20240507 02:08:16.078758 9950 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240507 02:08:16.079308 9964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:08:16.079469 9950 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240507 02:08:16.079533 9950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-2/wal
uuid: "5c7b3b793c584ec4a1446f92ea69b09d"
format_stamp: "Formatted at 2024-05-07 02:08:16 on dist-test-slave-v0bg"
server_key: "f60388f088beeaaa618df3f2bd9f384c"
server_key_iv: "4c6e9651699499897691095a7ab22a5a"
server_key_version: "encryptionkey@0"
I20240507 02:08:16.079587 9950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:08:16.082656 9950 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:08:16.082804 9950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:08:16.082890 9950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:08:16.083024 9950 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:08:16.083238 9950 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:08:16.083266 9950 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:08:16.083289 9950 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:08:16.083314 9950 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:08:16.087922 9950 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.67:44113
I20240507 02:08:16.087988 10077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.67:44113 every 8 connection(s)
I20240507 02:08:16.088186 9950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
I20240507 02:08:16.090018 9667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:16.088735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.67:40949 (local address 127.31.131.126:36577)
0507 02:08:16.088784 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:16.088786 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:16.088954 (+ 168us) server_negotiation.cc:407] Connection header received
0507 02:08:16.089308 (+ 354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:16.089309 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:16.089313 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:16.089365 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:08:16.089855 (+ 490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:08:16.089856 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:08:16.089856 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0507 02:08:16.089859 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:08:16.089871 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0507 02:08:16.089874 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:08:16.089913 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0507 02:08:16.089959 (+ 46us) server_negotiation.cc:299] Negotiation successful
0507 02:08:16.089968 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20240507 02:08:16.090121 10080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:16.088702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:36577 (local address 127.31.131.67:40949)
0507 02:08:16.088855 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0507 02:08:16.088880 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0507 02:08:16.089240 (+ 360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:08:16.089394 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:08:16.089397 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:08:16.089404 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:08:16.089769 (+ 365us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0507 02:08:16.089769 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:08:16.089780 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0507 02:08:16.089795 (+ 15us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:08:16.089911 (+ 116us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:08:16.089913 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:08:16.089914 (+ 1us) client_negotiation.cc:769] Sending connection context
0507 02:08:16.089972 (+ 58us) client_negotiation.cc:240] Negotiation successful
0507 02:08:16.090002 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":63,"threads_started":1}
I20240507 02:08:16.090618 10078 heartbeater.cc:347] Connected to a master server at 127.31.131.126:36577
I20240507 02:08:16.090692 10078 heartbeater.cc:464] Registering TS with master...
I20240507 02:08:16.090788 10078 heartbeater.cc:511] Master 127.31.131.126:36577 requested a full tablet report, sending...
I20240507 02:08:16.091096 9628 ts_manager.cc:194] Registered new tserver with Master: 5c7b3b793c584ec4a1446f92ea69b09d (127.31.131.67:44113)
I20240507 02:08:16.097568 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 9950
I20240507 02:08:16.097656 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1715047604443657-32269-0/minicluster-data/ts-2/wal/instance
I20240507 02:08:16.097798 32269 external_mini_cluster.cc:1428] Setting key dc29a2daa294c0804ba7d9d897b51266
I20240507 02:08:16.098466 32269 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20240507 02:08:16.099962 9667 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:08:16.099562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37742 (local address 127.31.131.126:36577)
0507 02:08:16.099614 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:16.099616 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:16.099689 (+ 73us) server_negotiation.cc:407] Connection header received
0507 02:08:16.099752 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:16.099753 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:16.099757 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:16.099797 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:08:16.099912 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37742: BlockingRecv error: recv got EOF from 127.0.0.1:37742 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
I20240507 02:08:16.100003 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 9688
I20240507 02:08:16.103794 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 9819
I20240507 02:08:16.107213 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 9950
I20240507 02:08:16.110296 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 9597
2024-05-07T02:08:16Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (482 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 07 02:08:16 dist-test-slave-v0bg krb5kdc[10088](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 07 02:08:16 dist-test-slave-v0bg krb5kdc[10088](info): set up 2 sockets
May 07 02:08:16 dist-test-slave-v0bg krb5kdc[10088](info): commencing operation
krb5kdc: starting...
W20240507 02:08:16.999392 32269 mini_kdc.cc:121] Time spent starting KDC: real 0.871s user 0.002s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/krb5kdc/test-user.keytab.
May 07 02:08:17 dist-test-slave-v0bg krb5kdc[10088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047697, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-07T02:08:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-07T02:08:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.131.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab.
Entry for principal kudu/127.31.131.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab.
I20240507 02:08:17.097028 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:38237
--webserver_interface=127.31.131.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab
--principal=kudu/127.31.131.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:42617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.131.126:38237
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:08:17.169013 10101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:08:17.169173 10101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:08:17.169201 10101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:08:17.170837 10101 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240507 02:08:17.170877 10101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:08:17.170895 10101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:08:17.170909 10101 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240507 02:08:17.170924 10101 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240507 02:08:17.172423 10101 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:42617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.131.126:38237
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab
--principal=kudu/127.31.131.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:38237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.131.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:08:17.172608 10101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:08:17.172755 10101 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:08:17.176182 10107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:08:17.176280 10101 server_base.cc:1034] running on GCE node
W20240507 02:08:17.176182 10106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:08:17.176388 10109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:08:17.176591 10101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:08:17.176793 10101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:08:17.177927 10101 hybrid_clock.cc:648] HybridClock initialized: now 1715047697177918 us; error 34 us; skew 500 ppm
May 07 02:08:17 dist-test-slave-v0bg krb5kdc[10088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047697, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:08:17.180083 10101 init.cc:377] Logged in from keytab as kudu/127.31.131.126@KRBTEST.COM (short username kudu)
I20240507 02:08:17.180421 10101 webserver.cc:460] Webserver started at http://127.31.131.126:42497/ using document root <none> and password file <none>
I20240507 02:08:17.180547 10101 fs_manager.cc:359] Metadata directory not provided
I20240507 02:08:17.180578 10101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:08:17.180648 10101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:08:17.181448 10101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/minicluster-data/master-0/data/instance:
uuid: "aab484eda4aa4a1589aea2064d2fe8d9"
format_stamp: "Formatted at 2024-05-07 02:08:17 on dist-test-slave-v0bg"
server_key: "a81d3da862758921849e475603d3e4d5"
server_key_iv: "ad1902cde5de7e53925bd2aca612e6ab"
server_key_version: "encryptionkey@0"
I20240507 02:08:17.181674 10101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/minicluster-data/master-0/wal/instance:
uuid: "aab484eda4aa4a1589aea2064d2fe8d9"
format_stamp: "Formatted at 2024-05-07 02:08:17 on dist-test-slave-v0bg"
server_key: "a81d3da862758921849e475603d3e4d5"
server_key_iv: "ad1902cde5de7e53925bd2aca612e6ab"
server_key_version: "encryptionkey@0"
I20240507 02:08:17.182638 10101 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20240507 02:08:17.183192 10116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:08:17.183331 10101 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240507 02:08:17.183390 10101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/minicluster-data/master-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/minicluster-data/master-0/wal
uuid: "aab484eda4aa4a1589aea2064d2fe8d9"
format_stamp: "Formatted at 2024-05-07 02:08:17 on dist-test-slave-v0bg"
server_key: "a81d3da862758921849e475603d3e4d5"
server_key_iv: "ad1902cde5de7e53925bd2aca612e6ab"
server_key_version: "encryptionkey@0"
I20240507 02:08:17.183449 10101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:08:17.197625 10101 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:08:17.199056 10101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:08:17.199144 10101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:08:17.202153 10101 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.126:38237
I20240507 02:08:17.202180 10168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.126:38237 every 8 connection(s)
I20240507 02:08:17.202423 10101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
I20240507 02:08:17.202972 10169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:08:17.204922 10169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aab484eda4aa4a1589aea2064d2fe8d9: Bootstrap starting.
I20240507 02:08:17.205354 10169 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aab484eda4aa4a1589aea2064d2fe8d9: Neither blocks nor log segments found. Creating new log.
I20240507 02:08:17.205543 10169 log.cc:824] T 00000000000000000000000000000000 P aab484eda4aa4a1589aea2064d2fe8d9: Log is configured to *not* fsync() on all Append() calls
I20240507 02:08:17.206058 10169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aab484eda4aa4a1589aea2064d2fe8d9: No bootstrap required, opened a new log
I20240507 02:08:17.207340 10169 raft_consensus.cc:348] T 00000000000000000000000000000000 P aab484eda4aa4a1589aea2064d2fe8d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aab484eda4aa4a1589aea2064d2fe8d9" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 38237 } }
I20240507 02:08:17.207440 10169 raft_consensus.cc:374] T 00000000000000000000000000000000 P aab484eda4aa4a1589aea2064d2fe8d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:08:17.207456 10169 raft_consensus.cc:729] T 00000000000000000000000000000000 P aab484eda4aa4a1589aea2064d2fe8d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aab484eda4aa4a1589aea2064d2fe8d9, State: Initialized, Role: FOLLOWER
I20240507 02:08:17.207530 10169 consensus_queue.cc:260] T 00000000000000000000000000000000 P aab484eda4aa4a1589aea2064d2fe8d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aab484eda4aa4a1589aea2064d2fe8d9" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 38237 } }
I20240507 02:08:17.207576 10169 raft_consensus.cc:388] T 00000000000000000000000000000000 P aab484eda4aa4a1589aea2064d2fe8d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 02:08:17.207602 10169 raft_consensus.cc:482] T 00000000000000000000000000000000 P aab484eda4aa4a1589aea2064d2fe8d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 02:08:17.207633 10169 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aab484eda4aa4a1589aea2064d2fe8d9 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:08:17.208086 10169 raft_consensus.cc:504] T 00000000000000000000000000000000 P aab484eda4aa4a1589aea2064d2fe8d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aab484eda4aa4a1589aea2064d2fe8d9" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 38237 } }
I20240507 02:08:17.208168 10169 leader_election.cc:302] T 00000000000000000000000000000000 P aab484eda4aa4a1589aea2064d2fe8d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aab484eda4aa4a1589aea2064d2fe8d9; no voters:
I20240507 02:08:17.208288 10169 leader_election.cc:288] T 00000000000000000000000000000000 P aab484eda4aa4a1589aea2064d2fe8d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240507 02:08:17.208334 10172 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aab484eda4aa4a1589aea2064d2fe8d9 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:08:17.208474 10169 sys_catalog.cc:564] T 00000000000000000000000000000000 P aab484eda4aa4a1589aea2064d2fe8d9 [sys.catalog]: configured and running, proceeding with master startup.
I20240507 02:08:17.208498 10172 raft_consensus.cc:686] T 00000000000000000000000000000000 P aab484eda4aa4a1589aea2064d2fe8d9 [term 1 LEADER]: Becoming Leader. State: Replica: aab484eda4aa4a1589aea2064d2fe8d9, State: Running, Role: LEADER
I20240507 02:08:17.208565 10172 consensus_queue.cc:237] T 00000000000000000000000000000000 P aab484eda4aa4a1589aea2064d2fe8d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aab484eda4aa4a1589aea2064d2fe8d9" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 38237 } }
I20240507 02:08:17.208861 10174 sys_catalog.cc:455] T 00000000000000000000000000000000 P aab484eda4aa4a1589aea2064d2fe8d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aab484eda4aa4a1589aea2064d2fe8d9. Latest consensus state: current_term: 1 leader_uuid: "aab484eda4aa4a1589aea2064d2fe8d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aab484eda4aa4a1589aea2064d2fe8d9" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 38237 } } }
I20240507 02:08:17.208891 10173 sys_catalog.cc:455] T 00000000000000000000000000000000 P aab484eda4aa4a1589aea2064d2fe8d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aab484eda4aa4a1589aea2064d2fe8d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aab484eda4aa4a1589aea2064d2fe8d9" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 38237 } } }
I20240507 02:08:17.208964 10173 sys_catalog.cc:458] T 00000000000000000000000000000000 P aab484eda4aa4a1589aea2064d2fe8d9 [sys.catalog]: This master's current role is: LEADER
I20240507 02:08:17.209129 10174 sys_catalog.cc:458] T 00000000000000000000000000000000 P aab484eda4aa4a1589aea2064d2fe8d9 [sys.catalog]: This master's current role is: LEADER
I20240507 02:08:17.209143 10176 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 02:08:17.209771 10176 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240507 02:08:17.209918 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 10101
I20240507 02:08:17.210016 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/minicluster-data/master-0/wal/instance
I20240507 02:08:17.210136 32269 external_mini_cluster.cc:1428] Setting key 82371782485fa30baeb46d7c29f9ceff
I20240507 02:08:17.214097 10176 catalog_manager.cc:1347] Generated new cluster ID: c1a4383bcc844f73b8517aff4b628464
I20240507 02:08:17.214154 10176 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
May 07 02:08:17 dist-test-slave-v0bg krb5kdc[10088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047697, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:08:17.219049 10191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:17.210461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53620 (local address 127.31.131.126:38237)
0507 02:08:17.210602 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:17.210611 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:17.210762 (+ 151us) server_negotiation.cc:407] Connection header received
0507 02:08:17.210941 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:17.210946 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:17.210987 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:17.211120 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:08:17.211974 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:17.212698 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:08:17.213452 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:17.213618 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:17.215841 (+ 2223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:08:17.215847 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:08:17.215847 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:08:17.215863 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:08:17.217750 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:08:17.218092 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:08:17.218094 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:08:17.218095 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:08:17.218145 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:08:17.218281 (+ 136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:08:17.218282 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:08:17.218283 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:08:17.218490 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:08:17.218640 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0507 02:08:17.218732 (+ 92us) server_negotiation.cc:299] Negotiation successful
0507 02:08:17.218843 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":65,"threads_started":1}
I20240507 02:08:17.223075 10176 catalog_manager.cc:1370] Generated new certificate authority record
I20240507 02:08:17.223515 10176 catalog_manager.cc:1504] Loading token signing keys...
I20240507 02:08:17.227962 10176 catalog_manager.cc:5857] T 00000000000000000000000000000000 P aab484eda4aa4a1589aea2064d2fe8d9: Generated new TSK 0
I20240507 02:08:17.228085 10176 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.131.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab.
Entry for principal kudu/127.31.131.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab.
I20240507 02:08:17.252857 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.65:0
--local_ip_for_outbound_sockets=127.31.131.65
--webserver_interface=127.31.131.65
--webserver_port=0
--tserver_master_addrs=127.31.131.126:38237
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab
--principal=kudu/127.31.131.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:42617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:08:17.326332 10195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:08:17.326514 10195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:08:17.326563 10195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:08:17.327808 10195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:08:17.327853 10195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:08:17.327889 10195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.65
I20240507 02:08:17.329280 10195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:42617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab
--principal=kudu/127.31.131.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.131.65
--webserver_port=0
--tserver_master_addrs=127.31.131.126:38237
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.65
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:08:17.329478 10195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:08:17.329641 10195 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:08:17.333618 10200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:08:17.333632 10201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:08:17.333873 10195 server_base.cc:1034] running on GCE node
W20240507 02:08:17.333710 10203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:08:17.334131 10195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:08:17.334347 10195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:08:17.335513 10195 hybrid_clock.cc:648] HybridClock initialized: now 1715047697335503 us; error 28 us; skew 500 ppm
May 07 02:08:17 dist-test-slave-v0bg krb5kdc[10088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047697, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:08:17.338070 10195 init.cc:377] Logged in from keytab as kudu/127.31.131.65@KRBTEST.COM (short username kudu)
I20240507 02:08:17.338479 10195 webserver.cc:460] Webserver started at http://127.31.131.65:38499/ using document root <none> and password file <none>
I20240507 02:08:17.338672 10195 fs_manager.cc:359] Metadata directory not provided
I20240507 02:08:17.338717 10195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:08:17.338810 10195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:08:17.339869 10195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/minicluster-data/ts-0/data/instance:
uuid: "6f39eb47f8294fd2a774fc7a4d0b0c6c"
format_stamp: "Formatted at 2024-05-07 02:08:17 on dist-test-slave-v0bg"
server_key: "93d52a756b0167ea05e408af61758bbb"
server_key_iv: "18990876326d2804dc6ea73adb2e2771"
server_key_version: "encryptionkey@0"
I20240507 02:08:17.340173 10195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/minicluster-data/ts-0/wal/instance:
uuid: "6f39eb47f8294fd2a774fc7a4d0b0c6c"
format_stamp: "Formatted at 2024-05-07 02:08:17 on dist-test-slave-v0bg"
server_key: "93d52a756b0167ea05e408af61758bbb"
server_key_iv: "18990876326d2804dc6ea73adb2e2771"
server_key_version: "encryptionkey@0"
I20240507 02:08:17.341442 10195 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240507 02:08:17.342103 10210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:08:17.342248 10195 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240507 02:08:17.342320 10195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/minicluster-data/ts-0/wal
uuid: "6f39eb47f8294fd2a774fc7a4d0b0c6c"
format_stamp: "Formatted at 2024-05-07 02:08:17 on dist-test-slave-v0bg"
server_key: "93d52a756b0167ea05e408af61758bbb"
server_key_iv: "18990876326d2804dc6ea73adb2e2771"
server_key_version: "encryptionkey@0"
I20240507 02:08:17.342376 10195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:08:17.365725 10195 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:08:17.367152 10195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:08:17.367238 10195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:08:17.367372 10195 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:08:17.367580 10195 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:08:17.367610 10195 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:08:17.367631 10195 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:08:17.367650 10195 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:08:17.373000 10195 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.65:35021
I20240507 02:08:17.373054 10323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.65:35021 every 8 connection(s)
I20240507 02:08:17.373283 10195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
I20240507 02:08:17.376411 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 10195
I20240507 02:08:17.376528 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/minicluster-data/ts-0/wal/instance
I20240507 02:08:17.376957 32269 external_mini_cluster.cc:1428] Setting key b9ff005f412b4dc02fce22854b5fa191
May 07 02:08:17 dist-test-slave-v0bg krb5kdc[10088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047697, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.65@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:08:17.380935 10191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:17.373865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:51747 (local address 127.31.131.126:38237)
0507 02:08:17.373916 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:17.373918 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:17.374055 (+ 137us) server_negotiation.cc:407] Connection header received
0507 02:08:17.374477 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:17.374478 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:17.374490 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:17.374540 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:08:17.375425 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:17.375883 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:08:17.376430 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:17.376608 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:17.378400 (+ 1792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:08:17.378403 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:08:17.378404 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:08:17.378409 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:08:17.379809 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:08:17.380156 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:08:17.380157 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:08:17.380157 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:08:17.380180 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:08:17.380339 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:08:17.380340 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:08:17.380340 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:08:17.380445 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:08:17.380511 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0507 02:08:17.380744 (+ 233us) server_negotiation.cc:299] Negotiation successful
0507 02:08:17.380836 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20240507 02:08:17.381004 10326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:17.373848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:38237 (local address 127.31.131.65:51747)
0507 02:08:17.373968 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0507 02:08:17.373993 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0507 02:08:17.374392 (+ 399us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:08:17.374605 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:08:17.374608 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:08:17.374841 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:08:17.375330 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:17.375334 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:17.375956 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:08:17.375956 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:08:17.376331 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:17.376333 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:17.376456 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:17.376884 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:08:17.376903 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:08:17.378295 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:08:17.379896 (+ 1601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:08:17.379902 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:08:17.379904 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:08:17.380084 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:08:17.380238 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:08:17.380239 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:08:17.380239 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:08:17.380282 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:08:17.380505 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:08:17.380508 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:08:17.380599 (+ 91us) client_negotiation.cc:769] Sending connection context
0507 02:08:17.380710 (+ 111us) client_negotiation.cc:240] Negotiation successful
0507 02:08:17.380834 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":59,"threads_started":1}
I20240507 02:08:17.381484 10324 heartbeater.cc:347] Connected to a master server at 127.31.131.126:38237
I20240507 02:08:17.381564 10324 heartbeater.cc:464] Registering TS with master...
I20240507 02:08:17.381734 10324 heartbeater.cc:511] Master 127.31.131.126:38237 requested a full tablet report, sending...
I20240507 02:08:17.382193 10133 ts_manager.cc:194] Registered new tserver with Master: 6f39eb47f8294fd2a774fc7a4d0b0c6c (127.31.131.65:35021)
I20240507 02:08:17.383152 10133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.131.65@KRBTEST.COM'} at 127.31.131.65:51747
WARNING: no policy specified for kudu/127.31.131.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab.
Entry for principal kudu/127.31.131.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab.
I20240507 02:08:17.398041 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.66:0
--local_ip_for_outbound_sockets=127.31.131.66
--webserver_interface=127.31.131.66
--webserver_port=0
--tserver_master_addrs=127.31.131.126:38237
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab
--principal=kudu/127.31.131.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:42617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:08:17.470711 10329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:08:17.470867 10329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:08:17.470899 10329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:08:17.472177 10329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:08:17.472222 10329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:08:17.472256 10329 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.66
I20240507 02:08:17.473631 10329 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:42617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab
--principal=kudu/127.31.131.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.131.66
--webserver_port=0
--tserver_master_addrs=127.31.131.126:38237
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10329
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.66
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:08:17.473812 10329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:08:17.473970 10329 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:08:17.477432 10334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:08:17.477523 10329 server_base.cc:1034] running on GCE node
W20240507 02:08:17.477459 10337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:08:17.477564 10335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:08:17.477780 10329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:08:17.477941 10329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:08:17.479058 10329 hybrid_clock.cc:648] HybridClock initialized: now 1715047697479047 us; error 28 us; skew 500 ppm
May 07 02:08:17 dist-test-slave-v0bg krb5kdc[10088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047697, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:08:17.481020 10329 init.cc:377] Logged in from keytab as kudu/127.31.131.66@KRBTEST.COM (short username kudu)
I20240507 02:08:17.481372 10329 webserver.cc:460] Webserver started at http://127.31.131.66:40013/ using document root <none> and password file <none>
I20240507 02:08:17.481527 10329 fs_manager.cc:359] Metadata directory not provided
I20240507 02:08:17.481563 10329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:08:17.481659 10329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:08:17.482476 10329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/minicluster-data/ts-1/data/instance:
uuid: "61af758b276a487292e8d96227b5ae4a"
format_stamp: "Formatted at 2024-05-07 02:08:17 on dist-test-slave-v0bg"
server_key: "d5f91e19f9d629bd26ce1037edde8fd4"
server_key_iv: "4a49683523faad0a1966f2ed96b54edc"
server_key_version: "encryptionkey@0"
I20240507 02:08:17.482764 10329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/minicluster-data/ts-1/wal/instance:
uuid: "61af758b276a487292e8d96227b5ae4a"
format_stamp: "Formatted at 2024-05-07 02:08:17 on dist-test-slave-v0bg"
server_key: "d5f91e19f9d629bd26ce1037edde8fd4"
server_key_iv: "4a49683523faad0a1966f2ed96b54edc"
server_key_version: "encryptionkey@0"
I20240507 02:08:17.483736 10329 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240507 02:08:17.484272 10344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:08:17.484408 10329 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240507 02:08:17.484472 10329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/minicluster-data/ts-1/wal
uuid: "61af758b276a487292e8d96227b5ae4a"
format_stamp: "Formatted at 2024-05-07 02:08:17 on dist-test-slave-v0bg"
server_key: "d5f91e19f9d629bd26ce1037edde8fd4"
server_key_iv: "4a49683523faad0a1966f2ed96b54edc"
server_key_version: "encryptionkey@0"
I20240507 02:08:17.484524 10329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:08:17.513938 10329 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:08:17.515318 10329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:08:17.515419 10329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:08:17.515563 10329 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:08:17.515789 10329 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:08:17.515817 10329 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:08:17.515841 10329 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:08:17.515858 10329 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:08:17.520541 10329 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.66:41095
I20240507 02:08:17.520592 10457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.66:41095 every 8 connection(s)
I20240507 02:08:17.520798 10329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
I20240507 02:08:17.521087 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 10329
I20240507 02:08:17.521189 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/minicluster-data/ts-1/wal/instance
I20240507 02:08:17.521329 32269 external_mini_cluster.cc:1428] Setting key ffd33433d3fc03970ce43a1dc7f4a5fe
May 07 02:08:17 dist-test-slave-v0bg krb5kdc[10088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047697, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.66@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.131.67@KRBTEST.COM; defaulting to no policy
I20240507 02:08:17.528352 10191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:17.521481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.66:38223 (local address 127.31.131.126:38237)
0507 02:08:17.521529 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:17.521532 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:17.521541 (+ 9us) server_negotiation.cc:407] Connection header received
0507 02:08:17.522011 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:17.522012 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:17.522022 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:17.522072 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:08:17.522960 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:17.523439 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:08:17.523935 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:17.524095 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:17.525983 (+ 1888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:08:17.525986 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:08:17.525986 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:08:17.525992 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:08:17.527261 (+ 1269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:08:17.527670 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:08:17.527671 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:08:17.527671 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:08:17.527686 (+ 15us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:08:17.527854 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:08:17.527854 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:08:17.527855 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:08:17.527924 (+ 69us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:08:17.527963 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0507 02:08:17.528230 (+ 267us) server_negotiation.cc:299] Negotiation successful
0507 02:08:17.528290 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20240507 02:08:17.528482 10460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:17.521310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:38237 (local address 127.31.131.66:38223)
0507 02:08:17.521444 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0507 02:08:17.521469 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0507 02:08:17.521917 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:08:17.522128 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:08:17.522132 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:08:17.522366 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:08:17.522850 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:17.522854 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:17.523515 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:08:17.523517 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:08:17.523866 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:17.523867 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:17.523986 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:17.524445 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:08:17.524457 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:08:17.525903 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:08:17.527337 (+ 1434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:08:17.527343 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:08:17.527345 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:08:17.527584 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:08:17.527736 (+ 152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:08:17.527737 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:08:17.527737 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:08:17.527806 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:08:17.527969 (+ 163us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:08:17.527972 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:08:17.528096 (+ 124us) client_negotiation.cc:769] Sending connection context
0507 02:08:17.528216 (+ 120us) client_negotiation.cc:240] Negotiation successful
0507 02:08:17.528330 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":54,"threads_started":1}
I20240507 02:08:17.528896 10458 heartbeater.cc:347] Connected to a master server at 127.31.131.126:38237
I20240507 02:08:17.528970 10458 heartbeater.cc:464] Registering TS with master...
I20240507 02:08:17.529105 10458 heartbeater.cc:511] Master 127.31.131.126:38237 requested a full tablet report, sending...
I20240507 02:08:17.529424 10133 ts_manager.cc:194] Registered new tserver with Master: 61af758b276a487292e8d96227b5ae4a (127.31.131.66:41095)
I20240507 02:08:17.529754 10133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.131.66@KRBTEST.COM'} at 127.31.131.66:38223
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab.
Entry for principal kudu/127.31.131.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab.
I20240507 02:08:17.544262 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.67:0
--local_ip_for_outbound_sockets=127.31.131.67
--webserver_interface=127.31.131.67
--webserver_port=0
--tserver_master_addrs=127.31.131.126:38237
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab
--principal=kudu/127.31.131.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:42617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:08:17.617733 10463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:08:17.617889 10463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:08:17.617910 10463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:08:17.619115 10463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:08:17.619153 10463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:08:17.619177 10463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.67
I20240507 02:08:17.620550 10463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:42617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab
--principal=kudu/127.31.131.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.131.67
--webserver_port=0
--tserver_master_addrs=127.31.131.126:38237
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.67
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:08:17.620697 10463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:08:17.620832 10463 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:08:17.624305 10469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:08:17.624315 10471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:08:17.624451 10468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:08:17.624490 10463 server_base.cc:1034] running on GCE node
I20240507 02:08:17.624637 10463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:08:17.624809 10463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:08:17.625936 10463 hybrid_clock.cc:648] HybridClock initialized: now 1715047697625921 us; error 31 us; skew 500 ppm
May 07 02:08:17 dist-test-slave-v0bg krb5kdc[10088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047697, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:08:17.627918 10463 init.cc:377] Logged in from keytab as kudu/127.31.131.67@KRBTEST.COM (short username kudu)
I20240507 02:08:17.628259 10463 webserver.cc:460] Webserver started at http://127.31.131.67:33769/ using document root <none> and password file <none>
I20240507 02:08:17.628412 10463 fs_manager.cc:359] Metadata directory not provided
I20240507 02:08:17.628449 10463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:08:17.628547 10463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:08:17.629364 10463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/minicluster-data/ts-2/data/instance:
uuid: "53251d80f8454fa79c6777293679f986"
format_stamp: "Formatted at 2024-05-07 02:08:17 on dist-test-slave-v0bg"
server_key: "59da380fea09070a448ddcb138a08e79"
server_key_iv: "71f7813336b0a75100503ed61ba297c8"
server_key_version: "encryptionkey@0"
I20240507 02:08:17.629632 10463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/minicluster-data/ts-2/wal/instance:
uuid: "53251d80f8454fa79c6777293679f986"
format_stamp: "Formatted at 2024-05-07 02:08:17 on dist-test-slave-v0bg"
server_key: "59da380fea09070a448ddcb138a08e79"
server_key_iv: "71f7813336b0a75100503ed61ba297c8"
server_key_version: "encryptionkey@0"
I20240507 02:08:17.630625 10463 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240507 02:08:17.631177 10478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:08:17.631311 10463 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240507 02:08:17.631387 10463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/minicluster-data/ts-2/wal
uuid: "53251d80f8454fa79c6777293679f986"
format_stamp: "Formatted at 2024-05-07 02:08:17 on dist-test-slave-v0bg"
server_key: "59da380fea09070a448ddcb138a08e79"
server_key_iv: "71f7813336b0a75100503ed61ba297c8"
server_key_version: "encryptionkey@0"
I20240507 02:08:17.631440 10463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:08:17.645000 10463 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:08:17.646340 10463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:08:17.646436 10463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:08:17.646607 10463 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:08:17.646843 10463 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:08:17.646872 10463 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:08:17.646896 10463 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:08:17.646911 10463 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:08:17.651650 10463 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.67:45379
I20240507 02:08:17.651701 10591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.67:45379 every 8 connection(s)
I20240507 02:08:17.651907 10463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
May 07 02:08:17 dist-test-slave-v0bg krb5kdc[10088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047697, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.67@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:08:17.657352 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 10463
I20240507 02:08:17.657474 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/minicluster-data/ts-2/wal/instance
I20240507 02:08:17.657629 32269 external_mini_cluster.cc:1428] Setting key 73f01225c0232d206ea7f69b128aa453
I20240507 02:08:17.658659 10191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:17.652419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.67:36507 (local address 127.31.131.126:38237)
0507 02:08:17.652457 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:17.652459 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:17.652593 (+ 134us) server_negotiation.cc:407] Connection header received
0507 02:08:17.652964 (+ 371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:17.652964 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:17.652972 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:17.653016 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:08:17.653780 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:17.654264 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:08:17.654749 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:17.654848 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:17.656375 (+ 1527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:08:17.656379 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:08:17.656379 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:08:17.656385 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:08:17.657504 (+ 1119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:08:17.657858 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:08:17.657859 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:08:17.657860 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:08:17.657884 (+ 24us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:08:17.658072 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:08:17.658073 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:08:17.658074 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:08:17.658174 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:08:17.658231 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0507 02:08:17.658480 (+ 249us) server_negotiation.cc:299] Negotiation successful
0507 02:08:17.658543 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20240507 02:08:17.658720 10594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:17.652392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:38237 (local address 127.31.131.67:36507)
0507 02:08:17.652514 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0507 02:08:17.652538 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0507 02:08:17.652893 (+ 355us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:08:17.653059 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:08:17.653062 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:08:17.653282 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:08:17.653718 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:17.653721 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:17.654322 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:08:17.654323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:08:17.654694 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:17.654695 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:17.654780 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:17.655134 (+ 354us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:08:17.655144 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:08:17.656289 (+ 1145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:08:17.657578 (+ 1289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:08:17.657583 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:08:17.657585 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:08:17.657799 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:08:17.657948 (+ 149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:08:17.657948 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:08:17.657949 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:08:17.658015 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:08:17.658233 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:08:17.658237 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:08:17.658355 (+ 118us) client_negotiation.cc:769] Sending connection context
0507 02:08:17.658458 (+ 103us) client_negotiation.cc:240] Negotiation successful
0507 02:08:17.658568 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":50,"threads_started":1}
I20240507 02:08:17.659076 10592 heartbeater.cc:347] Connected to a master server at 127.31.131.126:38237
I20240507 02:08:17.659149 10592 heartbeater.cc:464] Registering TS with master...
I20240507 02:08:17.659282 10592 heartbeater.cc:511] Master 127.31.131.126:38237 requested a full tablet report, sending...
I20240507 02:08:17.659591 10133 ts_manager.cc:194] Registered new tserver with Master: 53251d80f8454fa79c6777293679f986 (127.31.131.67:45379)
I20240507 02:08:17.659910 10133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.131.67@KRBTEST.COM'} at 127.31.131.67:36507
I20240507 02:08:17.669010 32269 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240507 02:08:17.674430 10191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:17.670049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53632 (local address 127.31.131.126:38237)
0507 02:08:17.670086 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:17.670087 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:17.670162 (+ 75us) server_negotiation.cc:407] Connection header received
0507 02:08:17.670206 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:17.670206 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:17.670213 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:17.670247 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:08:17.670866 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:17.671275 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:08:17.671823 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:17.671976 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:17.672700 (+ 724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:08:17.672703 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:08:17.672703 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:08:17.672708 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:08:17.673612 (+ 904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:08:17.673862 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:08:17.673863 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:08:17.673863 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:08:17.673881 (+ 18us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:08:17.674000 (+ 119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:08:17.674000 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:08:17.674000 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:08:17.674091 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:08:17.674185 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0507 02:08:17.674275 (+ 90us) server_negotiation.cc:299] Negotiation successful
0507 02:08:17.674339 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20240507 02:08:17.675691 10133 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53632:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 02:08:17.676916 10133 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 02:08:17.681509 10601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:17.678519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.65:35021 (local address 127.0.0.1:46340)
0507 02:08:17.678753 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0507 02:08:17.678769 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0507 02:08:17.678824 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:08:17.679035 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:08:17.679037 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:08:17.679039 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:08:17.679256 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:17.679258 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:17.680654 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:08:17.680655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:08:17.681225 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:17.681227 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:17.681286 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:17.681321 (+ 35us) client_negotiation.cc:769] Sending connection context
0507 02:08:17.681364 (+ 43us) client_negotiation.cc:240] Negotiation successful
0507 02:08:17.681402 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":73,"threads_started":1}
I20240507 02:08:17.681689 10602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:17.678537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.67:45379 (local address 127.0.0.1:35916)
0507 02:08:17.678922 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0507 02:08:17.678929 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0507 02:08:17.678982 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:08:17.679143 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:08:17.679144 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:08:17.679146 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:08:17.679309 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:17.679311 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:17.680489 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:08:17.680490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:08:17.681299 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:17.681300 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:17.681354 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:17.681357 (+ 3us) client_negotiation.cc:769] Sending connection context
0507 02:08:17.681386 (+ 29us) client_negotiation.cc:240] Negotiation successful
0507 02:08:17.681436 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":103,"threads_started":1}
I20240507 02:08:17.682137 10604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:17.678729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35916 (local address 127.31.131.67:45379)
0507 02:08:17.678870 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:17.678877 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:17.678993 (+ 116us) server_negotiation.cc:407] Connection header received
0507 02:08:17.679023 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:17.679024 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:17.679036 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:17.679101 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:08:17.679369 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:17.680422 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:08:17.681550 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:17.681983 (+ 433us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:17.682013 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0507 02:08:17.682038 (+ 25us) server_negotiation.cc:299] Negotiation successful
0507 02:08:17.682072 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":71,"threads_started":1}
I20240507 02:08:17.682155 10600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:17.678567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.66:41095 (local address 127.0.0.1:60720)
0507 02:08:17.679045 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0507 02:08:17.679052 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0507 02:08:17.679089 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:08:17.679265 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:08:17.679266 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:08:17.679267 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:08:17.679484 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:17.679486 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:17.680372 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:08:17.680373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:08:17.681230 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:17.681232 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:17.682000 (+ 768us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:17.682003 (+ 3us) client_negotiation.cc:769] Sending connection context
0507 02:08:17.682067 (+ 64us) client_negotiation.cc:240] Negotiation successful
0507 02:08:17.682103 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":64,"threads_started":1}
I20240507 02:08:17.682238 10605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:17.678731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60720 (local address 127.31.131.66:41095)
0507 02:08:17.678955 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:17.678962 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:17.679135 (+ 173us) server_negotiation.cc:407] Connection header received
0507 02:08:17.679169 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:17.679171 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:17.679186 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:17.679252 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:08:17.679605 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:17.680299 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:08:17.681423 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:17.681925 (+ 502us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:17.681969 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0507 02:08:17.682107 (+ 138us) server_negotiation.cc:299] Negotiation successful
0507 02:08:17.682160 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":71,"threads_started":1}
I20240507 02:08:17.682696 10603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:17.678608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46340 (local address 127.31.131.65:35021)
0507 02:08:17.678880 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:17.678886 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:17.678894 (+ 8us) server_negotiation.cc:407] Connection header received
0507 02:08:17.678930 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:17.678932 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:17.678943 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:17.679015 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:08:17.679338 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:17.680585 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:08:17.681324 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:17.682515 (+ 1191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:17.682569 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0507 02:08:17.682599 (+ 30us) server_negotiation.cc:299] Negotiation successful
0507 02:08:17.682638 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":76,"threads_started":1}
I20240507 02:08:17.682677 10526 tablet_service.cc:1467] Processing CreateTablet for tablet e4002540be024b8a9f670c7ef943328a (DEFAULT_TABLE table=test-table [id=eca6923b105945f085131d80ac44fd5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:08:17.682782 10392 tablet_service.cc:1467] Processing CreateTablet for tablet e4002540be024b8a9f670c7ef943328a (DEFAULT_TABLE table=test-table [id=eca6923b105945f085131d80ac44fd5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:08:17.682924 10526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4002540be024b8a9f670c7ef943328a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:08:17.682993 10392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4002540be024b8a9f670c7ef943328a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:08:17.683164 10258 tablet_service.cc:1467] Processing CreateTablet for tablet e4002540be024b8a9f670c7ef943328a (DEFAULT_TABLE table=test-table [id=eca6923b105945f085131d80ac44fd5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:08:17.683419 10258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4002540be024b8a9f670c7ef943328a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:08:17.685446 10606 tablet_bootstrap.cc:492] T e4002540be024b8a9f670c7ef943328a P 6f39eb47f8294fd2a774fc7a4d0b0c6c: Bootstrap starting.
I20240507 02:08:17.685850 10607 tablet_bootstrap.cc:492] T e4002540be024b8a9f670c7ef943328a P 53251d80f8454fa79c6777293679f986: Bootstrap starting.
I20240507 02:08:17.686136 10606 tablet_bootstrap.cc:654] T e4002540be024b8a9f670c7ef943328a P 6f39eb47f8294fd2a774fc7a4d0b0c6c: Neither blocks nor log segments found. Creating new log.
I20240507 02:08:17.686321 10607 tablet_bootstrap.cc:654] T e4002540be024b8a9f670c7ef943328a P 53251d80f8454fa79c6777293679f986: Neither blocks nor log segments found. Creating new log.
I20240507 02:08:17.686432 10606 log.cc:824] T e4002540be024b8a9f670c7ef943328a P 6f39eb47f8294fd2a774fc7a4d0b0c6c: Log is configured to *not* fsync() on all Append() calls
I20240507 02:08:17.686600 10607 log.cc:824] T e4002540be024b8a9f670c7ef943328a P 53251d80f8454fa79c6777293679f986: Log is configured to *not* fsync() on all Append() calls
I20240507 02:08:17.686722 10608 tablet_bootstrap.cc:492] T e4002540be024b8a9f670c7ef943328a P 61af758b276a487292e8d96227b5ae4a: Bootstrap starting.
I20240507 02:08:17.687139 10606 tablet_bootstrap.cc:492] T e4002540be024b8a9f670c7ef943328a P 6f39eb47f8294fd2a774fc7a4d0b0c6c: No bootstrap required, opened a new log
I20240507 02:08:17.687194 10606 ts_tablet_manager.cc:1367] T e4002540be024b8a9f670c7ef943328a P 6f39eb47f8294fd2a774fc7a4d0b0c6c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240507 02:08:17.687233 10607 tablet_bootstrap.cc:492] T e4002540be024b8a9f670c7ef943328a P 53251d80f8454fa79c6777293679f986: No bootstrap required, opened a new log
I20240507 02:08:17.687299 10607 ts_tablet_manager.cc:1367] T e4002540be024b8a9f670c7ef943328a P 53251d80f8454fa79c6777293679f986: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20240507 02:08:17.687361 10608 tablet_bootstrap.cc:654] T e4002540be024b8a9f670c7ef943328a P 61af758b276a487292e8d96227b5ae4a: Neither blocks nor log segments found. Creating new log.
I20240507 02:08:17.687635 10608 log.cc:824] T e4002540be024b8a9f670c7ef943328a P 61af758b276a487292e8d96227b5ae4a: Log is configured to *not* fsync() on all Append() calls
I20240507 02:08:17.688310 10608 tablet_bootstrap.cc:492] T e4002540be024b8a9f670c7ef943328a P 61af758b276a487292e8d96227b5ae4a: No bootstrap required, opened a new log
I20240507 02:08:17.688372 10608 ts_tablet_manager.cc:1367] T e4002540be024b8a9f670c7ef943328a P 61af758b276a487292e8d96227b5ae4a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240507 02:08:17.688679 10606 raft_consensus.cc:348] T e4002540be024b8a9f670c7ef943328a P 6f39eb47f8294fd2a774fc7a4d0b0c6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f39eb47f8294fd2a774fc7a4d0b0c6c" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35021 } } peers { permanent_uuid: "53251d80f8454fa79c6777293679f986" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 45379 } } peers { permanent_uuid: "61af758b276a487292e8d96227b5ae4a" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 41095 } }
I20240507 02:08:17.688779 10606 raft_consensus.cc:374] T e4002540be024b8a9f670c7ef943328a P 6f39eb47f8294fd2a774fc7a4d0b0c6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:08:17.688800 10606 raft_consensus.cc:729] T e4002540be024b8a9f670c7ef943328a P 6f39eb47f8294fd2a774fc7a4d0b0c6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f39eb47f8294fd2a774fc7a4d0b0c6c, State: Initialized, Role: FOLLOWER
I20240507 02:08:17.688896 10606 consensus_queue.cc:260] T e4002540be024b8a9f670c7ef943328a P 6f39eb47f8294fd2a774fc7a4d0b0c6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f39eb47f8294fd2a774fc7a4d0b0c6c" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35021 } } peers { permanent_uuid: "53251d80f8454fa79c6777293679f986" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 45379 } } peers { permanent_uuid: "61af758b276a487292e8d96227b5ae4a" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 41095 } }
I20240507 02:08:17.689106 10606 ts_tablet_manager.cc:1398] T e4002540be024b8a9f670c7ef943328a P 6f39eb47f8294fd2a774fc7a4d0b0c6c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240507 02:08:17.689126 10324 heartbeater.cc:503] Master 127.31.131.126:38237 was elected leader, sending a full tablet report...
I20240507 02:08:17.689332 10607 raft_consensus.cc:348] T e4002540be024b8a9f670c7ef943328a P 53251d80f8454fa79c6777293679f986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f39eb47f8294fd2a774fc7a4d0b0c6c" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35021 } } peers { permanent_uuid: "53251d80f8454fa79c6777293679f986" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 45379 } } peers { permanent_uuid: "61af758b276a487292e8d96227b5ae4a" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 41095 } }
I20240507 02:08:17.689468 10607 raft_consensus.cc:374] T e4002540be024b8a9f670c7ef943328a P 53251d80f8454fa79c6777293679f986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:08:17.689492 10607 raft_consensus.cc:729] T e4002540be024b8a9f670c7ef943328a P 53251d80f8454fa79c6777293679f986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53251d80f8454fa79c6777293679f986, State: Initialized, Role: FOLLOWER
I20240507 02:08:17.689570 10607 consensus_queue.cc:260] T e4002540be024b8a9f670c7ef943328a P 53251d80f8454fa79c6777293679f986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f39eb47f8294fd2a774fc7a4d0b0c6c" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35021 } } peers { permanent_uuid: "53251d80f8454fa79c6777293679f986" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 45379 } } peers { permanent_uuid: "61af758b276a487292e8d96227b5ae4a" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 41095 } }
I20240507 02:08:17.689790 10607 ts_tablet_manager.cc:1398] T e4002540be024b8a9f670c7ef943328a P 53251d80f8454fa79c6777293679f986: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240507 02:08:17.689882 10592 heartbeater.cc:503] Master 127.31.131.126:38237 was elected leader, sending a full tablet report...
I20240507 02:08:17.690357 10608 raft_consensus.cc:348] T e4002540be024b8a9f670c7ef943328a P 61af758b276a487292e8d96227b5ae4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f39eb47f8294fd2a774fc7a4d0b0c6c" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35021 } } peers { permanent_uuid: "53251d80f8454fa79c6777293679f986" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 45379 } } peers { permanent_uuid: "61af758b276a487292e8d96227b5ae4a" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 41095 } }
I20240507 02:08:17.690474 10608 raft_consensus.cc:374] T e4002540be024b8a9f670c7ef943328a P 61af758b276a487292e8d96227b5ae4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:08:17.690496 10608 raft_consensus.cc:729] T e4002540be024b8a9f670c7ef943328a P 61af758b276a487292e8d96227b5ae4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61af758b276a487292e8d96227b5ae4a, State: Initialized, Role: FOLLOWER
I20240507 02:08:17.690603 10608 consensus_queue.cc:260] T e4002540be024b8a9f670c7ef943328a P 61af758b276a487292e8d96227b5ae4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f39eb47f8294fd2a774fc7a4d0b0c6c" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35021 } } peers { permanent_uuid: "53251d80f8454fa79c6777293679f986" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 45379 } } peers { permanent_uuid: "61af758b276a487292e8d96227b5ae4a" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 41095 } }
I20240507 02:08:17.690814 10608 ts_tablet_manager.cc:1398] T e4002540be024b8a9f670c7ef943328a P 61af758b276a487292e8d96227b5ae4a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240507 02:08:17.690856 10458 heartbeater.cc:503] Master 127.31.131.126:38237 was elected leader, sending a full tablet report...
W20240507 02:08:17.771199 10459 tablet.cc:2369] T e4002540be024b8a9f670c7ef943328a P 61af758b276a487292e8d96227b5ae4a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240507 02:08:17.873844 10325 tablet.cc:2369] T e4002540be024b8a9f670c7ef943328a P 6f39eb47f8294fd2a774fc7a4d0b0c6c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240507 02:08:17.902326 10593 tablet.cc:2369] T e4002540be024b8a9f670c7ef943328a P 53251d80f8454fa79c6777293679f986: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240507 02:08:18.012992 10612 raft_consensus.cc:482] T e4002540be024b8a9f670c7ef943328a P 6f39eb47f8294fd2a774fc7a4d0b0c6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 02:08:18.013094 10612 raft_consensus.cc:504] T e4002540be024b8a9f670c7ef943328a P 6f39eb47f8294fd2a774fc7a4d0b0c6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f39eb47f8294fd2a774fc7a4d0b0c6c" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35021 } } peers { permanent_uuid: "53251d80f8454fa79c6777293679f986" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 45379 } } peers { permanent_uuid: "61af758b276a487292e8d96227b5ae4a" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 41095 } }
I20240507 02:08:18.013384 10612 leader_election.cc:288] T e4002540be024b8a9f670c7ef943328a P 6f39eb47f8294fd2a774fc7a4d0b0c6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53251d80f8454fa79c6777293679f986 (127.31.131.67:45379), 61af758b276a487292e8d96227b5ae4a (127.31.131.66:41095)
I20240507 02:08:18.015511 10616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:18.013542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.67:45379 (local address 127.31.131.65:57783)
0507 02:08:18.013655 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0507 02:08:18.013666 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0507 02:08:18.013739 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:08:18.013856 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:08:18.013857 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:08:18.013860 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:08:18.014075 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:18.014077 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:18.014719 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:08:18.014720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:08:18.015273 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:18.015274 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:18.015337 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:18.015340 (+ 3us) client_negotiation.cc:769] Sending connection context
0507 02:08:18.015366 (+ 26us) client_negotiation.cc:240] Negotiation successful
0507 02:08:18.015399 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":52,"threads_started":1}
I20240507 02:08:18.015509 10615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:18.013560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.66:41095 (local address 127.31.131.65:41331)
0507 02:08:18.013670 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0507 02:08:18.013679 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0507 02:08:18.013737 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:08:18.013856 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:08:18.013856 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:08:18.013858 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:08:18.014077 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:18.014079 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:18.014700 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:08:18.014701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:08:18.015258 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:18.015259 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:18.015318 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:18.015323 (+ 5us) client_negotiation.cc:769] Sending connection context
0507 02:08:18.015348 (+ 25us) client_negotiation.cc:240] Negotiation successful
0507 02:08:18.015382 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":53,"threads_started":1}
I20240507 02:08:18.015831 10605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:18.013598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:41331 (local address 127.31.131.66:41095)
0507 02:08:18.013654 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:18.013656 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:18.013737 (+ 81us) server_negotiation.cc:407] Connection header received
0507 02:08:18.013791 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:18.013792 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:18.013801 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:18.013845 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:08:18.014150 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:18.014629 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:08:18.015334 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:18.015718 (+ 384us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:18.015725 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0507 02:08:18.015743 (+ 18us) server_negotiation.cc:299] Negotiation successful
0507 02:08:18.015771 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20240507 02:08:18.015880 10604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:18.013621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:57783 (local address 127.31.131.67:45379)
0507 02:08:18.013671 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:18.013673 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:18.013721 (+ 48us) server_negotiation.cc:407] Connection header received
0507 02:08:18.013794 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:18.013795 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:18.013804 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:18.013843 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:08:18.014148 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:18.014636 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:08:18.015344 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:18.015763 (+ 419us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:18.015771 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0507 02:08:18.015793 (+ 22us) server_negotiation.cc:299] Negotiation successful
0507 02:08:18.015822 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20240507 02:08:18.016191 10412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4002540be024b8a9f670c7ef943328a" candidate_uuid: "6f39eb47f8294fd2a774fc7a4d0b0c6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61af758b276a487292e8d96227b5ae4a" is_pre_election: true
I20240507 02:08:18.016211 10546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4002540be024b8a9f670c7ef943328a" candidate_uuid: "6f39eb47f8294fd2a774fc7a4d0b0c6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53251d80f8454fa79c6777293679f986" is_pre_election: true
I20240507 02:08:18.016316 10412 raft_consensus.cc:2454] T e4002540be024b8a9f670c7ef943328a P 61af758b276a487292e8d96227b5ae4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f39eb47f8294fd2a774fc7a4d0b0c6c in term 0.
I20240507 02:08:18.016319 10546 raft_consensus.cc:2454] T e4002540be024b8a9f670c7ef943328a P 53251d80f8454fa79c6777293679f986 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f39eb47f8294fd2a774fc7a4d0b0c6c in term 0.
I20240507 02:08:18.016494 10214 leader_election.cc:302] T e4002540be024b8a9f670c7ef943328a P 6f39eb47f8294fd2a774fc7a4d0b0c6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53251d80f8454fa79c6777293679f986, 6f39eb47f8294fd2a774fc7a4d0b0c6c; no voters:
I20240507 02:08:18.016620 10612 raft_consensus.cc:2781] T e4002540be024b8a9f670c7ef943328a P 6f39eb47f8294fd2a774fc7a4d0b0c6c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 02:08:18.016677 10612 raft_consensus.cc:482] T e4002540be024b8a9f670c7ef943328a P 6f39eb47f8294fd2a774fc7a4d0b0c6c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 02:08:18.016705 10612 raft_consensus.cc:3037] T e4002540be024b8a9f670c7ef943328a P 6f39eb47f8294fd2a774fc7a4d0b0c6c [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:08:18.017270 10612 raft_consensus.cc:504] T e4002540be024b8a9f670c7ef943328a P 6f39eb47f8294fd2a774fc7a4d0b0c6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f39eb47f8294fd2a774fc7a4d0b0c6c" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35021 } } peers { permanent_uuid: "53251d80f8454fa79c6777293679f986" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 45379 } } peers { permanent_uuid: "61af758b276a487292e8d96227b5ae4a" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 41095 } }
I20240507 02:08:18.017379 10612 leader_election.cc:288] T e4002540be024b8a9f670c7ef943328a P 6f39eb47f8294fd2a774fc7a4d0b0c6c [CANDIDATE]: Term 1 election: Requested vote from peers 53251d80f8454fa79c6777293679f986 (127.31.131.67:45379), 61af758b276a487292e8d96227b5ae4a (127.31.131.66:41095)
I20240507 02:08:18.017552 10412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4002540be024b8a9f670c7ef943328a" candidate_uuid: "6f39eb47f8294fd2a774fc7a4d0b0c6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61af758b276a487292e8d96227b5ae4a"
I20240507 02:08:18.017553 10546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4002540be024b8a9f670c7ef943328a" candidate_uuid: "6f39eb47f8294fd2a774fc7a4d0b0c6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53251d80f8454fa79c6777293679f986"
I20240507 02:08:18.017614 10412 raft_consensus.cc:3037] T e4002540be024b8a9f670c7ef943328a P 61af758b276a487292e8d96227b5ae4a [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:08:18.017619 10546 raft_consensus.cc:3037] T e4002540be024b8a9f670c7ef943328a P 53251d80f8454fa79c6777293679f986 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:08:18.018250 10412 raft_consensus.cc:2454] T e4002540be024b8a9f670c7ef943328a P 61af758b276a487292e8d96227b5ae4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f39eb47f8294fd2a774fc7a4d0b0c6c in term 1.
I20240507 02:08:18.018260 10546 raft_consensus.cc:2454] T e4002540be024b8a9f670c7ef943328a P 53251d80f8454fa79c6777293679f986 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f39eb47f8294fd2a774fc7a4d0b0c6c in term 1.
I20240507 02:08:18.018380 10214 leader_election.cc:302] T e4002540be024b8a9f670c7ef943328a P 6f39eb47f8294fd2a774fc7a4d0b0c6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53251d80f8454fa79c6777293679f986, 6f39eb47f8294fd2a774fc7a4d0b0c6c; no voters:
I20240507 02:08:18.018464 10612 raft_consensus.cc:2781] T e4002540be024b8a9f670c7ef943328a P 6f39eb47f8294fd2a774fc7a4d0b0c6c [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:08:18.018626 10612 raft_consensus.cc:686] T e4002540be024b8a9f670c7ef943328a P 6f39eb47f8294fd2a774fc7a4d0b0c6c [term 1 LEADER]: Becoming Leader. State: Replica: 6f39eb47f8294fd2a774fc7a4d0b0c6c, State: Running, Role: LEADER
I20240507 02:08:18.018707 10612 consensus_queue.cc:237] T e4002540be024b8a9f670c7ef943328a P 6f39eb47f8294fd2a774fc7a4d0b0c6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f39eb47f8294fd2a774fc7a4d0b0c6c" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35021 } } peers { permanent_uuid: "53251d80f8454fa79c6777293679f986" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 45379 } } peers { permanent_uuid: "61af758b276a487292e8d96227b5ae4a" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 41095 } }
I20240507 02:08:18.019408 10133 catalog_manager.cc:5484] T e4002540be024b8a9f670c7ef943328a P 6f39eb47f8294fd2a774fc7a4d0b0c6c reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f39eb47f8294fd2a774fc7a4d0b0c6c (127.31.131.65). New cstate: current_term: 1 leader_uuid: "6f39eb47f8294fd2a774fc7a4d0b0c6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f39eb47f8294fd2a774fc7a4d0b0c6c" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53251d80f8454fa79c6777293679f986" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 45379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61af758b276a487292e8d96227b5ae4a" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 41095 } health_report { overall_health: UNKNOWN } } }
I20240507 02:08:18.033427 10603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:18.031867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46346 (local address 127.31.131.65:35021)
0507 02:08:18.031918 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:18.031920 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:18.031926 (+ 6us) server_negotiation.cc:407] Connection header received
0507 02:08:18.031956 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:18.031957 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:18.031969 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:18.032023 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0507 02:08:18.032206 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:18.032610 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:08:18.033083 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:18.033194 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:18.033226 (+ 32us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0507 02:08:18.033305 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0507 02:08:18.033359 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0507 02:08:18.033375 (+ 16us) server_negotiation.cc:299] Negotiation successful
0507 02:08:18.033389 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20240507 02:08:18.034806 10546 raft_consensus.cc:1261] T e4002540be024b8a9f670c7ef943328a P 53251d80f8454fa79c6777293679f986 [term 1 FOLLOWER]: Refusing update from remote peer 6f39eb47f8294fd2a774fc7a4d0b0c6c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 02:08:18.034806 10412 raft_consensus.cc:1261] T e4002540be024b8a9f670c7ef943328a P 61af758b276a487292e8d96227b5ae4a [term 1 FOLLOWER]: Refusing update from remote peer 6f39eb47f8294fd2a774fc7a4d0b0c6c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 02:08:18.035154 10612 consensus_queue.cc:1035] T e4002540be024b8a9f670c7ef943328a P 6f39eb47f8294fd2a774fc7a4d0b0c6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "61af758b276a487292e8d96227b5ae4a" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 41095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 02:08:18.035275 10617 consensus_queue.cc:1035] T e4002540be024b8a9f670c7ef943328a P 6f39eb47f8294fd2a774fc7a4d0b0c6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "53251d80f8454fa79c6777293679f986" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 45379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 02:08:18.037170 10620 mvcc.cc:205] Tried to move back new op lower bound from 7024835371148500992 to 7024835371085193216. Current Snapshot: MvccSnapshot[applied={T|T < 7024835371148500992}]
I20240507 02:08:18.037250 10622 mvcc.cc:205] Tried to move back new op lower bound from 7024835371148500992 to 7024835371085193216. Current Snapshot: MvccSnapshot[applied={T|T < 7024835371148500992}]
I20240507 02:08:18.037330 10621 mvcc.cc:205] Tried to move back new op lower bound from 7024835371148500992 to 7024835371085193216. Current Snapshot: MvccSnapshot[applied={T|T < 7024835371148500992}]
I20240507 02:08:18.039695 10132 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53632:
table { table_name: "test-table" } modify_external_catalogs: true
I20240507 02:08:18.039777 10132 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53632:
table { table_name: "test-table" } modify_external_catalogs: true
I20240507 02:08:18.040380 10132 catalog_manager.cc:5771] T 00000000000000000000000000000000 P aab484eda4aa4a1589aea2064d2fe8d9: Sending DeleteTablet for 3 replicas of tablet e4002540be024b8a9f670c7ef943328a
I20240507 02:08:18.040653 10258 tablet_service.cc:1514] Processing DeleteTablet for tablet e4002540be024b8a9f670c7ef943328a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:08:18 UTC) from {username='kudu', principal='kudu/127.31.131.126@KRBTEST.COM'} at 127.0.0.1:46340
I20240507 02:08:18.040668 10392 tablet_service.cc:1514] Processing DeleteTablet for tablet e4002540be024b8a9f670c7ef943328a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:08:18 UTC) from {username='kudu', principal='kudu/127.31.131.126@KRBTEST.COM'} at 127.0.0.1:60720
I20240507 02:08:18.040853 10526 tablet_service.cc:1514] Processing DeleteTablet for tablet e4002540be024b8a9f670c7ef943328a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:08:18 UTC) from {username='kudu', principal='kudu/127.31.131.126@KRBTEST.COM'} at 127.0.0.1:35916
I20240507 02:08:18.041061 10632 tablet_replica.cc:330] T e4002540be024b8a9f670c7ef943328a P 53251d80f8454fa79c6777293679f986: stopping tablet replica
I20240507 02:08:18.041111 10630 tablet_replica.cc:330] T e4002540be024b8a9f670c7ef943328a P 6f39eb47f8294fd2a774fc7a4d0b0c6c: stopping tablet replica
I20240507 02:08:18.041188 10632 raft_consensus.cc:2229] T e4002540be024b8a9f670c7ef943328a P 53251d80f8454fa79c6777293679f986 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 02:08:18.041231 10630 raft_consensus.cc:2229] T e4002540be024b8a9f670c7ef943328a P 6f39eb47f8294fd2a774fc7a4d0b0c6c [term 1 LEADER]: Raft consensus shutting down.
I20240507 02:08:18.041340 10632 raft_consensus.cc:2258] T e4002540be024b8a9f670c7ef943328a P 53251d80f8454fa79c6777293679f986 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 02:08:18.041348 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 10195
I20240507 02:08:18.041373 10630 raft_consensus.cc:2258] T e4002540be024b8a9f670c7ef943328a P 6f39eb47f8294fd2a774fc7a4d0b0c6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 02:08:18.042163 10632 ts_tablet_manager.cc:1875] T e4002540be024b8a9f670c7ef943328a P 53251d80f8454fa79c6777293679f986: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 02:08:18.042718 10631 tablet_replica.cc:330] T e4002540be024b8a9f670c7ef943328a P 61af758b276a487292e8d96227b5ae4a: stopping tablet replica
I20240507 02:08:18.042797 10631 raft_consensus.cc:2229] T e4002540be024b8a9f670c7ef943328a P 61af758b276a487292e8d96227b5ae4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 02:08:18.042877 10631 raft_consensus.cc:2258] T e4002540be024b8a9f670c7ef943328a P 61af758b276a487292e8d96227b5ae4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 02:08:18.043198 10631 ts_tablet_manager.cc:1875] T e4002540be024b8a9f670c7ef943328a P 61af758b276a487292e8d96227b5ae4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 02:08:18.043726 10632 ts_tablet_manager.cc:1888] T e4002540be024b8a9f670c7ef943328a P 53251d80f8454fa79c6777293679f986: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240507 02:08:18.043797 10632 log.cc:1196] T e4002540be024b8a9f670c7ef943328a P 53251d80f8454fa79c6777293679f986: Deleting WAL directory at /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/minicluster-data/ts-2/wal/wals/e4002540be024b8a9f670c7ef943328a
I20240507 02:08:18.044057 10632 ts_tablet_manager.cc:1909] T e4002540be024b8a9f670c7ef943328a P 53251d80f8454fa79c6777293679f986: Deleting consensus metadata
I20240507 02:08:18.044425 10631 ts_tablet_manager.cc:1888] T e4002540be024b8a9f670c7ef943328a P 61af758b276a487292e8d96227b5ae4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240507 02:08:18.044477 10631 log.cc:1196] T e4002540be024b8a9f670c7ef943328a P 61af758b276a487292e8d96227b5ae4a: Deleting WAL directory at /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1715047604443657-32269-0/minicluster-data/ts-1/wal/wals/e4002540be024b8a9f670c7ef943328a
I20240507 02:08:18.044667 10631 ts_tablet_manager.cc:1909] T e4002540be024b8a9f670c7ef943328a P 61af758b276a487292e8d96227b5ae4a: Deleting consensus metadata
I20240507 02:08:18.044965 10118 catalog_manager.cc:4830] TS 61af758b276a487292e8d96227b5ae4a (127.31.131.66:41095): tablet e4002540be024b8a9f670c7ef943328a (table test-table [id=eca6923b105945f085131d80ac44fd5c]) successfully deleted
W20240507 02:08:18.046119 10117 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.131.65:35021 (error 108)
I20240507 02:08:18.046478 10600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:18.046386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.65:35021 (local address 127.0.0.1:46358)
0507 02:08:18.046431 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0507 02:08:18.046456 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.65:35021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20240507 02:08:18.046587 10117 catalog_manager.cc:4573] TS 6f39eb47f8294fd2a774fc7a4d0b0c6c (127.31.131.65:35021): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e4002540be024b8a9f670c7ef943328a: Network error: Client connection negotiation failed: client connection to 127.31.131.65:35021: connect: Connection refused (error 111)
I20240507 02:08:18.046895 10120 catalog_manager.cc:4830] TS 53251d80f8454fa79c6777293679f986 (127.31.131.67:45379): tablet e4002540be024b8a9f670c7ef943328a (table test-table [id=eca6923b105945f085131d80ac44fd5c]) successfully deleted
I20240507 02:08:18.047286 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 10329
I20240507 02:08:18.051442 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 10463
I20240507 02:08:18.055344 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 10101
2024-05-07T02:08:18Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1945 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 07 02:08:18 dist-test-slave-v0bg krb5kdc[10636](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 07 02:08:18 dist-test-slave-v0bg krb5kdc[10636](info): set up 2 sockets
May 07 02:08:18 dist-test-slave-v0bg krb5kdc[10636](info): commencing operation
krb5kdc: starting...
W20240507 02:08:18.935425 32269 mini_kdc.cc:121] Time spent starting KDC: real 0.862s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/krb5kdc/test-user.keytab.
May 07 02:08:19 dist-test-slave-v0bg krb5kdc[10636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047699, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-07T02:08:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-07T02:08:19Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.31.131.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.31.131.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.31.131.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/krb5kdc/rangeradmin_127.31.131.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.31.131.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/krb5kdc/rangeradmin_127.31.131.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.31.131.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.31.131.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.31.131.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/krb5kdc/rangerlookup_127.31.131.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.31.131.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/krb5kdc/rangerlookup_127.31.131.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.31.131.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.31.131.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.31.131.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/krb5kdc/HTTP_127.31.131.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.31.131.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/krb5kdc/HTTP_127.31.131.84@KRBTEST.COM.keytab.
I20240507 02:08:19.070458 32269 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskTztKno/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/postgres -l logfile start
2024-05-07 02:08:21.240 UTC [10666] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
2024-05-07 02:08:21.240 UTC [10666] LOG: listening on IPv4 address "127.31.131.84", port 60667
2024-05-07 02:08:21.243 UTC [10666] LOG: listening on Unix socket "/tmp/.s.PGSQL.60667"
2024-05-07 02:08:21.261 UTC [10669] LOG: database system was shut down at 2024-05-07 02:08:20 UTC
2024-05-07 02:08:21.264 UTC [10666] LOG: database system is ready to accept connections
I20240507 02:08:22.079177 32269 mini_postgres.cc:96] Postgres bound to 60667
2024-05-07 02:08:22.084 UTC [10677] FATAL: database "slave" does not exist
127.31.131.84:60667 - accepting connections
I20240507 02:08:22.084652 32269 mini_ranger.cc:161] Starting Ranger...
I20240507 02:08:22.097064 32269 mini_ranger.cc:84] Created miniranger Postgres user
I20240507 02:08:22.474753 32269 mini_ranger.cc:87] Created ranger Postgres database
I20240507 02:08:22.474815 32269 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/ranger-admin
2024-05-07 02:08:22,671 [I] DB FLAVOR :POSTGRES
2024-05-07 02:08:22,672 [I] --------- Verifying Ranger DB connection ---------
2024-05-07 02:08:22,672 [I] Checking connection..
2024-05-07 02:08:22,672 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTztKno/build/release/bin/postgresql.jar:/tmp/dist-test-taskTztKno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.131.84:60667/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2024-05-07 02:08:22,904 [I] Checking connection passed.
2024-05-07 02:08:22,905 [I] --------- Verifying version history table ---------
2024-05-07 02:08:22,905 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTztKno/build/release/bin/postgresql.jar:/tmp/dist-test-taskTztKno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.131.84:60667/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2024-05-07 02:08:23,125 [I] Table x_db_version_h does not exist in database ranger
2024-05-07 02:08:23,125 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTztKno/build/release/bin/postgresql.jar:/tmp/dist-test-taskTztKno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.131.84:60667/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2024-05-07 02:08:23,362 [I] Table x_db_version_h does not exist in database ranger
2024-05-07 02:08:23,362 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2024-05-07 02:08:23,362 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTztKno/build/release/bin/postgresql.jar:/tmp/dist-test-taskTztKno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.131.84:60667/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskTztKno/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2024-05-07 02:08:23.606 UTC [10741] WARNING: there is no transaction in progress
2024-05-07 02:08:23,611 [I] create_dbversion_catalog.sql file imported successfully
2024-05-07 02:08:23,612 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTztKno/build/release/bin/postgresql.jar:/tmp/dist-test-taskTztKno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.131.84:60667/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2024-05-07 02:08:23,850 [I] Table x_db_version_h already exists in database 'ranger'
2024-05-07 02:08:23,851 [I] --------- Importing Ranger Core DB Schema ---------
2024-05-07 02:08:23,851 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTztKno/build/release/bin/postgresql.jar:/tmp/dist-test-taskTztKno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.131.84:60667/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2024-05-07 02:08:24,087 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTztKno/build/release/bin/postgresql.jar:/tmp/dist-test-taskTztKno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.131.84:60667/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2024-05-07 02:08:24,324 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTztKno/build/release/bin/postgresql.jar:/tmp/dist-test-taskTztKno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.131.84:60667/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-v0bg.c.gcp-upstream.internal','N') ;"
2024-05-07 02:08:24,548 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTztKno/build/release/bin/postgresql.jar:/tmp/dist-test-taskTztKno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.131.84:60667/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2024-05-07 02:08:24,765 [I] Table x_portal_user does not exist in database ranger
2024-05-07 02:08:24,765 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTztKno/build/release/bin/postgresql.jar:/tmp/dist-test-taskTztKno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.131.84:60667/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2024-05-07 02:08:24,981 [I] Table x_policy_ref_group does not exist in database ranger
2024-05-07 02:08:24,981 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTztKno/build/release/bin/postgresql.jar:/tmp/dist-test-taskTztKno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.131.84:60667/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2024-05-07 02:08:25,214 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTztKno/build/release/bin/postgresql.jar:/tmp/dist-test-taskTztKno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.131.84:60667/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2024-05-07 02:08:25,437 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2024-05-07 02:08:25,437 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTztKno/build/release/bin/postgresql.jar:/tmp/dist-test-taskTztKno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.131.84:60667/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskTztKno/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2024-05-07 02:08:26.278 UTC [10849] WARNING: there is no transaction in progress
2024-05-07 02:08:26.293 UTC [10849] WARNING: there is no transaction in progress
2024-05-07 02:08:26.308 UTC [10849] WARNING: there is no transaction in progress
2024-05-07 02:08:26.322 UTC [10849] WARNING: there is no transaction in progress
2024-05-07 02:08:26.335 UTC [10849] WARNING: there is no transaction in progress
2024-05-07 02:08:26.350 UTC [10849] WARNING: there is no transaction in progress
2024-05-07 02:08:26.398 UTC [10849] WARNING: there is no transaction in progress
2024-05-07 02:08:26.404 UTC [10849] WARNING: there is no transaction in progress
2024-05-07 02:08:26.416 UTC [10849] WARNING: there is no transaction in progress
2024-05-07 02:08:26.425 UTC [10849] WARNING: there is no transaction in progress
2024-05-07 02:08:26.433 UTC [10849] WARNING: there is no transaction in progress
2024-05-07 02:08:26.445 UTC [10849] WARNING: there is no transaction in progress
2024-05-07 02:08:26.453 UTC [10849] WARNING: there is no transaction in progress
2024-05-07 02:08:26.459 UTC [10849] WARNING: there is no transaction in progress
2024-05-07 02:08:26.465 UTC [10849] WARNING: there is no transaction in progress
2024-05-07 02:08:26.471 UTC [10849] WARNING: there is no transaction in progress
2024-05-07 02:08:27,013 [I] ranger_core_db_postgres.sql file imported successfully
2024-05-07 02:08:27,014 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTztKno/build/release/bin/postgresql.jar:/tmp/dist-test-taskTztKno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.131.84:60667/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2024-05-07 02:08:27,235 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2024-05-07 02:08:27,236 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTztKno/build/release/bin/postgresql.jar:/tmp/dist-test-taskTztKno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.131.84:60667/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2024-05-07 02:08:27,444 [I] Table x_portal_user already exists in database 'ranger'
2024-05-07 02:08:27,445 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTztKno/build/release/bin/postgresql.jar:/tmp/dist-test-taskTztKno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.131.84:60667/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2024-05-07 02:08:27,678 [I] Table x_policy_ref_group already exists in database 'ranger'
2024-05-07 02:08:27,679 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTztKno/build/release/bin/postgresql.jar:/tmp/dist-test-taskTztKno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.131.84:60667/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2024-05-07 02:08:27,910 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTztKno/build/release/bin/postgresql.jar:/tmp/dist-test-taskTztKno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.131.84:60667/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2024-05-07 02:08:28,147 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTztKno/build/release/bin/postgresql.jar:/tmp/dist-test-taskTztKno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.131.84:60667/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-v0bg.c.gcp-upstream.internal';"
2024-05-07 02:08:28,374 [I] CORE_DB_SCHEMA import status has been updated
2024-05-07 02:08:28,375 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTztKno/build/release/bin/postgresql.jar:/tmp/dist-test-taskTztKno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.131.84:60667/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2024-05-07 02:08:28,607 [I] DB_PATCHES have already been applied
I20240507 02:08:28.612205 32269 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/ranger-admin:/tmp/dist-test-taskTztKno/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskTztKno/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskTztKno/build/release/bin/postgresql.jar:/tmp/dist-test-taskTztKno/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20240507 02:08:28.612260 32269 mini_ranger.cc:193] Using host: 127.31.131.84
I20240507 02:08:28.614578 32269 mini_ranger.cc:227] Ranger admin URL: http://127.31.131.84:33123
May 07, 2024 2:08:28 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
May 07, 2024 2:08:28 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
May 07, 2024 2:08:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/ranger-admin/ews/webapp
May 07, 2024 2:08:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
May 07, 2024 2:08:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/ranger-admin/ews/webapp] .....
May 07, 2024 2:08:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/ranger-admin/ews/webapp].
May 07, 2024 2:08:29 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20240507 02:08:29.544072 32269 mini_ranger.cc:160] Time spent starting Ranger: real 7.459s user 0.000s sys 0.007s
May 07, 2024 2:08:47 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
May 07, 2024 2:08:47 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.AssetREST
May 07, 2024 2:08:47 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
May 07, 2024 2:08:47 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
May 07, 2024 2:08:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
May 07, 2024 2:08:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
May 07, 2024 2:08:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
May 07, 2024 2:08:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
May 07, 2024 2:08:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
May 07, 2024 2:08:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
May 07, 2024 2:08:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
May 07, 2024 2:08:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
May 07, 2024 2:08:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
May 07, 2024 2:08:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
May 07, 2024 2:08:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
May 07, 2024 2:08:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
May 07, 2024 2:08:47 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
May 07, 2024 2:08:47 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20240507 02:08:48.452941 32269 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.31.131.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.31.131.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.31.131.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/krb5kdc/rangerkms_127.31.131.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.31.131.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/krb5kdc/rangerkms_127.31.131.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.31.131.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.31.131.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.31.131.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/krb5kdc/HTTP_127.31.131.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.31.131.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/krb5kdc/HTTP_127.31.131.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
May 07 02:08:48 dist-test-slave-v0bg krb5kdc[10636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047728, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20240507 02:08:48.518751 32269 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20240507 02:08:48.525781 32269 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20240507 02:08:49.224812 32269 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20240507 02:08:49.224890 32269 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/ranger-kms
I20240507 02:08:49.224905 32269 mini_ranger_kms.cc:227] Using postgres at 127.31.131.84:60667
2024-05-07 02:08:49,379 [I] DB FLAVOR :POSTGRES
2024-05-07 02:08:49,379 [I] --------- Verifying Ranger DB connection ---------
2024-05-07 02:08:49,379 [I] Checking connection
2024-05-07 02:08:49,379 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTztKno/build/release/bin/postgresql.jar:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.131.84:60667/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2024-05-07 02:08:49,617 [I] connection success
2024-05-07 02:08:49,617 [I] --------- Verifying Ranger DB tables ---------
2024-05-07 02:08:49,617 [I] Verifying table ranger_masterkey in database rangerkms
2024-05-07 02:08:49,617 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTztKno/build/release/bin/postgresql.jar:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.131.84:60667/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2024-05-07 02:08:49,854 [I] Table ranger_masterkey does not exist in database rangerkms
2024-05-07 02:08:49,854 [I] --------- Importing Ranger Core DB Schema ---------
2024-05-07 02:08:49,854 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2024-05-07 02:08:49,854 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTztKno/build/release/bin/postgresql.jar:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.131.84:60667/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2024-05-07 02:08:50,109 [I] kms_core_db_postgres.sql DB schema imported successfully
I20240507 02:08:50.512526 32269 mini_ranger_kms.cc:310] Created kms service
I20240507 02:08:50.688354 32269 mini_ranger_kms.cc:326] Created kudu user
I20240507 02:08:50.726989 32269 mini_ranger_kms.cc:343] Created rangerkms user
May 07 02:08:50 dist-test-slave-v0bg krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047728, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.31.131.84@KRBTEST.COM
I20240507 02:08:51.082908 32269 mini_ranger_kms.cc:384] Added ranger policy
I20240507 02:08:51.083022 32269 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/ranger-kms:/tmp/dist-test-taskTztKno/build/release/bin/postgresql.jar:/tmp/dist-test-taskTztKno/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskTztKno/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20240507 02:08:51.083046 32269 mini_ranger_kms.cc:240] Using host: 127.31.131.84
I20240507 02:08:51.086028 32269 mini_ranger_kms.cc:276] Ranger KMS PID: 11029
I20240507 02:08:51.086107 32269 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.31.131.84:48073
May 07, 2024 2:08:51 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
May 07, 2024 2:08:51 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
May 07, 2024 2:08:52 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
May 07, 2024 2:08:52 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/ranger-kms/ews/webapp] .....
May 07, 2024 2:08:52 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/ranger-kms/ews/webapp].
May 07, 2024 2:08:52 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20240507 02:08:52.983420 32269 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.465s user 0.001s sys 0.137s
I20240507 02:08:52.983531 32269 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20240507 02:08:52.983578 32269 mini_ranger_kms.cc:401] 127.31.131.84:48073/kms/v1/keys
May 07 02:08:53 dist-test-slave-v0bg krb5kdc[10636](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1715047733, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.131.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
May 07 02:08:53 dist-test-slave-v0bg krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047733, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.131.84@KRBTEST.COM for HTTP/127.31.131.84@KRBTEST.COM
May 07 02:08:53 dist-test-slave-v0bg krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047733, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.131.84@KRBTEST.COM for HTTP/127.31.131.84@KRBTEST.COM
May 07 02:08:55 dist-test-slave-v0bg krb5kdc[10636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047735, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.31.131.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab.
Entry for principal kudu/127.31.131.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab.
I20240507 02:08:55.905109 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:35385
--webserver_interface=127.31.131.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab
--principal=kudu/127.31.131.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:37521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.131.126:35385
--ranger_config_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.131.84:48073/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:08:55.981380 11080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:08:55.981531 11080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:08:55.981549 11080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:08:55.982784 11080 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240507 02:08:55.982821 11080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:08:55.982833 11080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:08:55.982843 11080 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240507 02:08:55.982852 11080 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240507 02:08:55.984891 11080 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:37521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/master-0/wal
--ranger_kms_url=127.31.131.84:48073/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.31.131.126:35385
--ranger_config_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab
--principal=kudu/127.31.131.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:35385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.131.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:08:55.985088 11080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:08:55.985237 11080 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240507 02:08:55.988920 11080 server_base.cc:1034] running on GCE node
W20240507 02:08:55.988955 11085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:08:55.989078 11088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:08:55.989279 11086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:08:55.989480 11080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:08:55.989681 11080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:08:55.990789 11080 hybrid_clock.cc:648] HybridClock initialized: now 1715047735990779 us; error 30 us; skew 500 ppm
May 07 02:08:55 dist-test-slave-v0bg krb5kdc[10636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047735, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:08:55.996273 11080 init.cc:377] Logged in from keytab as kudu/127.31.131.126@KRBTEST.COM (short username kudu)
I20240507 02:08:55.996692 11080 webserver.cc:460] Webserver started at http://127.31.131.126:46161/ using document root <none> and password file <none>
I20240507 02:08:55.996877 11080 fs_manager.cc:359] Metadata directory not provided
I20240507 02:08:55.996928 11080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:08:55.997036 11080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
May 07 02:08:55 dist-test-slave-v0bg krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047735, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.126@KRBTEST.COM for HTTP/127.31.131.84@KRBTEST.COM
I20240507 02:08:56.044692 11080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/master-0/data/instance:
uuid: "d6d81953e1e74463b26017e31a910fa9"
format_stamp: "Formatted at 2024-05-07 02:08:56 on dist-test-slave-v0bg"
server_key: "5d814c3cd7806f92ec65dd61970d41a2"
server_key_iv: "1647a220f574c01be36d078a23f9846c"
server_key_version: "kuduclusterkey@0"
I20240507 02:08:56.045127 11080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/master-0/wal/instance:
uuid: "d6d81953e1e74463b26017e31a910fa9"
format_stamp: "Formatted at 2024-05-07 02:08:56 on dist-test-slave-v0bg"
server_key: "5d814c3cd7806f92ec65dd61970d41a2"
server_key_iv: "1647a220f574c01be36d078a23f9846c"
server_key_version: "kuduclusterkey@0"
I20240507 02:08:56.046384 11080 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20240507 02:08:56.059641 11095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:08:56.062598 11080 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.000s
I20240507 02:08:56.062685 11080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/master-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/master-0/wal
uuid: "d6d81953e1e74463b26017e31a910fa9"
format_stamp: "Formatted at 2024-05-07 02:08:56 on dist-test-slave-v0bg"
server_key: "5d814c3cd7806f92ec65dd61970d41a2"
server_key_iv: "1647a220f574c01be36d078a23f9846c"
server_key_version: "kuduclusterkey@0"
I20240507 02:08:56.062749 11080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:08:56.081781 11080 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:08:56.083537 11080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:08:56.083635 11080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:08:56.087473 11080 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.126:35385
I20240507 02:08:56.087514 11147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.126:35385 every 8 connection(s)
I20240507 02:08:56.087750 11080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
I20240507 02:08:56.088207 11148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:08:56.088996 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 11080
I20240507 02:08:56.089089 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/master-0/wal/instance
I20240507 02:08:56.089237 32269 external_mini_cluster.cc:1428] Setting key 77ab6616fdaa45b8c64ff74bbd276b88
I20240507 02:08:56.090258 11148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6d81953e1e74463b26017e31a910fa9: Bootstrap starting.
I20240507 02:08:56.090747 11148 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6d81953e1e74463b26017e31a910fa9: Neither blocks nor log segments found. Creating new log.
I20240507 02:08:56.090946 11148 log.cc:824] T 00000000000000000000000000000000 P d6d81953e1e74463b26017e31a910fa9: Log is configured to *not* fsync() on all Append() calls
I20240507 02:08:56.091676 11148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6d81953e1e74463b26017e31a910fa9: No bootstrap required, opened a new log
I20240507 02:08:56.093694 11148 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6d81953e1e74463b26017e31a910fa9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6d81953e1e74463b26017e31a910fa9" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 35385 } }
I20240507 02:08:56.093822 11148 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6d81953e1e74463b26017e31a910fa9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:08:56.093848 11148 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6d81953e1e74463b26017e31a910fa9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6d81953e1e74463b26017e31a910fa9, State: Initialized, Role: FOLLOWER
I20240507 02:08:56.093961 11148 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6d81953e1e74463b26017e31a910fa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6d81953e1e74463b26017e31a910fa9" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 35385 } }
I20240507 02:08:56.094044 11148 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6d81953e1e74463b26017e31a910fa9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 02:08:56.094097 11148 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6d81953e1e74463b26017e31a910fa9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 02:08:56.094144 11148 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6d81953e1e74463b26017e31a910fa9 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:08:56.094730 11148 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6d81953e1e74463b26017e31a910fa9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6d81953e1e74463b26017e31a910fa9" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 35385 } }
I20240507 02:08:56.094820 11148 leader_election.cc:302] T 00000000000000000000000000000000 P d6d81953e1e74463b26017e31a910fa9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6d81953e1e74463b26017e31a910fa9; no voters:
I20240507 02:08:56.094955 11148 leader_election.cc:288] T 00000000000000000000000000000000 P d6d81953e1e74463b26017e31a910fa9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240507 02:08:56.095054 11153 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6d81953e1e74463b26017e31a910fa9 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:08:56.095149 11148 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6d81953e1e74463b26017e31a910fa9 [sys.catalog]: configured and running, proceeding with master startup.
I20240507 02:08:56.095240 11153 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6d81953e1e74463b26017e31a910fa9 [term 1 LEADER]: Becoming Leader. State: Replica: d6d81953e1e74463b26017e31a910fa9, State: Running, Role: LEADER
I20240507 02:08:56.095356 11153 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6d81953e1e74463b26017e31a910fa9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6d81953e1e74463b26017e31a910fa9" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 35385 } }
I20240507 02:08:56.095652 11153 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6d81953e1e74463b26017e31a910fa9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6d81953e1e74463b26017e31a910fa9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6d81953e1e74463b26017e31a910fa9" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 35385 } } }
I20240507 02:08:56.095881 11153 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6d81953e1e74463b26017e31a910fa9 [sys.catalog]: This master's current role is: LEADER
I20240507 02:08:56.096033 11153 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6d81953e1e74463b26017e31a910fa9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6d81953e1e74463b26017e31a910fa9. Latest consensus state: current_term: 1 leader_uuid: "d6d81953e1e74463b26017e31a910fa9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6d81953e1e74463b26017e31a910fa9" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 35385 } } }
I20240507 02:08:56.096132 11153 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6d81953e1e74463b26017e31a910fa9 [sys.catalog]: This master's current role is: LEADER
I20240507 02:08:56.096269 11162 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 02:08:56.096469 11148 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20240507 02:08:56.096742 11162 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240507 02:08:56.098656 11162 catalog_manager.cc:1347] Generated new cluster ID: 4f2af23c72554e12a9ae0e88adea8ab3
I20240507 02:08:56.098704 11162 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
May 07 02:08:56 dist-test-slave-v0bg krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047735, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:08:56.111980 11151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:56.089625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44518 (local address 127.31.131.126:35385)
0507 02:08:56.089776 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:56.089786 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:56.089804 (+ 18us) server_negotiation.cc:407] Connection header received
0507 02:08:56.090181 (+ 377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:56.090185 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:56.090222 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:56.090343 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:08:56.094614 (+ 4271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:56.095828 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:08:56.097028 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:56.097267 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:56.104870 (+ 7603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:08:56.104990 (+ 120us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:08:56.104991 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:08:56.105077 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:08:56.108001 (+ 2924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:08:56.110621 (+ 2620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:08:56.110624 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:08:56.110624 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:08:56.110670 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:08:56.110856 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:08:56.110857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:08:56.110857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:08:56.111080 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:08:56.111604 (+ 524us) server_negotiation.cc:1035] Waiting for connection context
0507 02:08:56.111675 (+ 71us) server_negotiation.cc:299] Negotiation successful
0507 02:08:56.111805 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":76,"threads_started":1}
I20240507 02:08:56.145192 11162 catalog_manager.cc:1370] Generated new certificate authority record
I20240507 02:08:56.145671 11162 catalog_manager.cc:1504] Loading token signing keys...
I20240507 02:08:56.151609 11162 catalog_manager.cc:5857] T 00000000000000000000000000000000 P d6d81953e1e74463b26017e31a910fa9: Generated new TSK 0
I20240507 02:08:56.152072 11162 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.131.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab.
Entry for principal kudu/127.31.131.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab.
I20240507 02:08:56.781461 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.131.84:48073/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.65:0
--local_ip_for_outbound_sockets=127.31.131.65
--webserver_interface=127.31.131.65
--webserver_port=0
--tserver_master_addrs=127.31.131.126:35385
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab
--principal=kudu/127.31.131.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:37521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:08:56.878434 11193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:08:56.878597 11193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:08:56.878626 11193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:08:56.880414 11193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:08:56.880470 11193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:08:56.880504 11193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.65
I20240507 02:08:56.882584 11193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:37521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.31.131.84:48073/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab
--principal=kudu/127.31.131.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.131.65
--webserver_port=0
--tserver_master_addrs=127.31.131.126:35385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.65
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:08:56.882741 11193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:08:56.882891 11193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:08:56.886288 11198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:08:56.886301 11201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:08:56.886427 11193 server_base.cc:1034] running on GCE node
W20240507 02:08:56.886524 11199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:08:56.886842 11193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:08:56.887051 11193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:08:56.888175 11193 hybrid_clock.cc:648] HybridClock initialized: now 1715047736888163 us; error 32 us; skew 500 ppm
May 07 02:08:56 dist-test-slave-v0bg krb5kdc[10636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047736, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:08:56.890408 11193 init.cc:377] Logged in from keytab as kudu/127.31.131.65@KRBTEST.COM (short username kudu)
I20240507 02:08:56.890807 11193 webserver.cc:460] Webserver started at http://127.31.131.65:42409/ using document root <none> and password file <none>
I20240507 02:08:56.890990 11193 fs_manager.cc:359] Metadata directory not provided
I20240507 02:08:56.891031 11193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:08:56.891136 11193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
May 07 02:08:56 dist-test-slave-v0bg krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047736, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.65@KRBTEST.COM for HTTP/127.31.131.84@KRBTEST.COM
I20240507 02:08:56.907816 11193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/ts-0/data/instance:
uuid: "83d42e907cec44108f095074562ae6ac"
format_stamp: "Formatted at 2024-05-07 02:08:56 on dist-test-slave-v0bg"
server_key: "00ec13b69ec8299fdf678a6951020644"
server_key_iv: "e79476cb803306dae9a99f5b43182af7"
server_key_version: "kuduclusterkey@0"
I20240507 02:08:56.908217 11193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/ts-0/wal/instance:
uuid: "83d42e907cec44108f095074562ae6ac"
format_stamp: "Formatted at 2024-05-07 02:08:56 on dist-test-slave-v0bg"
server_key: "00ec13b69ec8299fdf678a6951020644"
server_key_iv: "e79476cb803306dae9a99f5b43182af7"
server_key_version: "kuduclusterkey@0"
I20240507 02:08:56.910589 11193 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20240507 02:08:56.922274 11208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:08:56.922435 11193 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20240507 02:08:56.922505 11193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/ts-0/wal
uuid: "83d42e907cec44108f095074562ae6ac"
format_stamp: "Formatted at 2024-05-07 02:08:56 on dist-test-slave-v0bg"
server_key: "00ec13b69ec8299fdf678a6951020644"
server_key_iv: "e79476cb803306dae9a99f5b43182af7"
server_key_version: "kuduclusterkey@0"
I20240507 02:08:56.922588 11193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:08:56.936712 11193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:08:56.938326 11193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:08:56.938421 11193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:08:56.938601 11193 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:08:56.938884 11193 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:08:56.938952 11193 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:08:56.938997 11193 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:08:56.939028 11193 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:08:56.945384 11193 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.65:39857
I20240507 02:08:56.945652 11193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
I20240507 02:08:56.946070 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 11193
I20240507 02:08:56.946167 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/ts-0/wal/instance
I20240507 02:08:56.946313 32269 external_mini_cluster.cc:1428] Setting key 2ac6399cb4e203b5f54da0437b282c6e
I20240507 02:08:56.947877 11323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.65:39857 every 8 connection(s)
May 07 02:08:56 dist-test-slave-v0bg krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047736, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.65@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:08:56.955605 11326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:56.946791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:39203 (local address 127.31.131.126:35385)
0507 02:08:56.947376 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:56.947378 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:56.948314 (+ 936us) server_negotiation.cc:407] Connection header received
0507 02:08:56.949191 (+ 877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:56.949192 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:56.949203 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:56.949293 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:08:56.949834 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:56.950407 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:08:56.951034 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:56.951202 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:56.953263 (+ 2061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:08:56.953267 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:08:56.953267 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:08:56.953273 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:08:56.954540 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:08:56.954901 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:08:56.954902 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:08:56.954903 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:08:56.954927 (+ 24us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:08:56.955073 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:08:56.955074 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:08:56.955075 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:08:56.955154 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:08:56.955206 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0507 02:08:56.955421 (+ 215us) server_negotiation.cc:299] Negotiation successful
0507 02:08:56.955496 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":74,"threads_started":1}
I20240507 02:08:56.955657 11327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:56.946961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:35385 (local address 127.31.131.65:39203)
0507 02:08:56.948194 (+ 1233us) negotiation.cc:107] Waiting for socket to connect
0507 02:08:56.948221 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0507 02:08:56.949082 (+ 861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:08:56.949361 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:08:56.949364 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:08:56.949504 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:08:56.949750 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:56.949753 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:56.950535 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:08:56.950536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:08:56.950919 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:56.950921 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:56.951077 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:56.951575 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:08:56.951588 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:08:56.953159 (+ 1571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:08:56.954657 (+ 1498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:08:56.954662 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:08:56.954664 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:08:56.954840 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:08:56.954978 (+ 138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:08:56.954978 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:08:56.954979 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:08:56.955027 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:08:56.955202 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:08:56.955206 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:08:56.955312 (+ 106us) client_negotiation.cc:769] Sending connection context
0507 02:08:56.955396 (+ 84us) client_negotiation.cc:240] Negotiation successful
0507 02:08:56.955498 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1201,"thread_start_us":1099,"threads_started":1}
I20240507 02:08:56.956265 11324 heartbeater.cc:347] Connected to a master server at 127.31.131.126:35385
I20240507 02:08:56.956353 11324 heartbeater.cc:464] Registering TS with master...
I20240507 02:08:56.956517 11324 heartbeater.cc:511] Master 127.31.131.126:35385 requested a full tablet report, sending...
I20240507 02:08:56.956950 11112 ts_manager.cc:194] Registered new tserver with Master: 83d42e907cec44108f095074562ae6ac (127.31.131.65:39857)
WARNING: no policy specified for kudu/127.31.131.66@KRBTEST.COM; defaulting to no policy
I20240507 02:08:56.957607 11112 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.131.65@KRBTEST.COM'} at 127.31.131.65:39203
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab.
Entry for principal kudu/127.31.131.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab.
I20240507 02:08:56.973251 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.131.84:48073/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.66:0
--local_ip_for_outbound_sockets=127.31.131.66
--webserver_interface=127.31.131.66
--webserver_port=0
--tserver_master_addrs=127.31.131.126:35385
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab
--principal=kudu/127.31.131.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:37521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:08:57.071355 11330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:08:57.071502 11330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:08:57.071527 11330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:08:57.072762 11330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:08:57.072798 11330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:08:57.072829 11330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.66
I20240507 02:08:57.074316 11330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:37521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.31.131.84:48073/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab
--principal=kudu/127.31.131.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.131.66
--webserver_port=0
--tserver_master_addrs=127.31.131.126:35385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.66
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:08:57.074509 11330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:08:57.074714 11330 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:08:57.078132 11337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:08:57.078162 11340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:08:57.078289 11338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:08:57.078377 11330 server_base.cc:1034] running on GCE node
I20240507 02:08:57.078588 11330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:08:57.078780 11330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:08:57.086077 11330 hybrid_clock.cc:648] HybridClock initialized: now 1715047737086057 us; error 30 us; skew 500 ppm
May 07 02:08:57 dist-test-slave-v0bg krb5kdc[10636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047737, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:08:57.088207 11330 init.cc:377] Logged in from keytab as kudu/127.31.131.66@KRBTEST.COM (short username kudu)
I20240507 02:08:57.088562 11330 webserver.cc:460] Webserver started at http://127.31.131.66:41063/ using document root <none> and password file <none>
I20240507 02:08:57.088716 11330 fs_manager.cc:359] Metadata directory not provided
I20240507 02:08:57.088752 11330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:08:57.088845 11330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
May 07 02:08:57 dist-test-slave-v0bg krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047737, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.66@KRBTEST.COM for HTTP/127.31.131.84@KRBTEST.COM
I20240507 02:08:57.105497 11330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/ts-1/data/instance:
uuid: "d968304608064d47a46e2d6da3b33106"
format_stamp: "Formatted at 2024-05-07 02:08:57 on dist-test-slave-v0bg"
server_key: "af2a4291706003061ba2c9ccf4490e6c"
server_key_iv: "29a4f54b01b008d5a7269732ac2be7fc"
server_key_version: "kuduclusterkey@0"
I20240507 02:08:57.105865 11330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/ts-1/wal/instance:
uuid: "d968304608064d47a46e2d6da3b33106"
format_stamp: "Formatted at 2024-05-07 02:08:57 on dist-test-slave-v0bg"
server_key: "af2a4291706003061ba2c9ccf4490e6c"
server_key_iv: "29a4f54b01b008d5a7269732ac2be7fc"
server_key_version: "kuduclusterkey@0"
I20240507 02:08:57.107381 11330 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
May 07 02:08:57 dist-test-slave-v0bg krb5kdc[10636](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1715047737, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:08:57.122820 11347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:08:57.125721 11330 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.001s sys 0.000s
I20240507 02:08:57.125811 11330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/ts-1/wal
uuid: "d968304608064d47a46e2d6da3b33106"
format_stamp: "Formatted at 2024-05-07 02:08:57 on dist-test-slave-v0bg"
server_key: "af2a4291706003061ba2c9ccf4490e6c"
server_key_iv: "29a4f54b01b008d5a7269732ac2be7fc"
server_key_version: "kuduclusterkey@0"
I20240507 02:08:57.125880 11330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:08:57.140571 11330 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:08:57.142746 11330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:08:57.142886 11330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:08:57.143070 11330 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:08:57.143404 11330 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:08:57.143452 11330 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:08:57.143515 11330 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:08:57.143563 11330 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:08:57.150025 11330 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.66:44837
I20240507 02:08:57.150413 11330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
I20240507 02:08:57.155489 11460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.66:44837 every 8 connection(s)
I20240507 02:08:57.158185 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 11330
I20240507 02:08:57.158282 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/ts-1/wal/instance
I20240507 02:08:57.158432 32269 external_mini_cluster.cc:1428] Setting key 850068bb5a4a292c3188e3e6de632446
WARNING: no policy specified for kudu/127.31.131.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.67@KRBTEST.COM" created.
May 07 02:08:57 dist-test-slave-v0bg krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047737, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.66@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:08:57.179867 11326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:57.156031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.66:40243 (local address 127.31.131.126:35385)
0507 02:08:57.156099 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:57.156101 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:57.166736 (+ 10635us) server_negotiation.cc:407] Connection header received
0507 02:08:57.167330 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:57.167331 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:57.167342 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:57.167369 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:08:57.168205 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:57.168994 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:08:57.169772 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:57.169964 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:57.176944 (+ 6980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:08:57.176947 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:08:57.176948 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:08:57.176954 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:08:57.178518 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:08:57.178969 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:08:57.178971 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:08:57.178971 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:08:57.179000 (+ 29us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:08:57.179201 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:08:57.179202 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:08:57.179202 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:08:57.179311 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:08:57.179352 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0507 02:08:57.179678 (+ 326us) server_negotiation.cc:299] Negotiation successful
0507 02:08:57.179768 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20240507 02:08:57.180272 11463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:57.156007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:35385 (local address 127.31.131.66:40243)
0507 02:08:57.166630 (+ 10623us) negotiation.cc:107] Waiting for socket to connect
0507 02:08:57.166661 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0507 02:08:57.167199 (+ 538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:08:57.167526 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:08:57.167530 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:08:57.167727 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:08:57.168100 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:57.168105 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:57.169100 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:08:57.169101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:08:57.169681 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:57.169683 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:57.169813 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:57.174331 (+ 4518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:08:57.174351 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:08:57.176843 (+ 2492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:08:57.178640 (+ 1797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:08:57.178646 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:08:57.178648 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:08:57.178882 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:08:57.179064 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:08:57.179065 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:08:57.179066 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:08:57.179138 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:08:57.179395 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:08:57.179400 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:08:57.179532 (+ 132us) client_negotiation.cc:769] Sending connection context
0507 02:08:57.179996 (+ 464us) client_negotiation.cc:240] Negotiation successful
0507 02:08:57.180095 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10586,"thread_start_us":59,"threads_started":1}
I20240507 02:08:57.180714 11461 heartbeater.cc:347] Connected to a master server at 127.31.131.126:35385
I20240507 02:08:57.180801 11461 heartbeater.cc:464] Registering TS with master...
I20240507 02:08:57.181002 11461 heartbeater.cc:511] Master 127.31.131.126:35385 requested a full tablet report, sending...
I20240507 02:08:57.181382 11112 ts_manager.cc:194] Registered new tserver with Master: d968304608064d47a46e2d6da3b33106 (127.31.131.66:44837)
I20240507 02:08:57.181727 11112 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.131.66@KRBTEST.COM'} at 127.31.131.66:40243
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab.
Entry for principal kudu/127.31.131.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab.
I20240507 02:08:57.190140 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.131.84:48073/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.67:0
--local_ip_for_outbound_sockets=127.31.131.67
--webserver_interface=127.31.131.67
--webserver_port=0
--tserver_master_addrs=127.31.131.126:35385
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab
--principal=kudu/127.31.131.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:37521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:08:57.294468 11466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:08:57.294761 11466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:08:57.294850 11466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:08:57.296811 11466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:08:57.296918 11466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:08:57.296994 11466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.67
I20240507 02:08:57.299232 11466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:37521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.31.131.84:48073/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab
--principal=kudu/127.31.131.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.131.67
--webserver_port=0
--tserver_master_addrs=127.31.131.126:35385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.67
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:08:57.299635 11466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:08:57.299865 11466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:08:57.304996 11472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:08:57.305187 11475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:08:57.305953 11466 server_base.cc:1034] running on GCE node
W20240507 02:08:57.306180 11473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:08:57.306396 11466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:08:57.306605 11466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:08:57.307755 11466 hybrid_clock.cc:648] HybridClock initialized: now 1715047737307739 us; error 30 us; skew 500 ppm
May 07 02:08:57 dist-test-slave-v0bg krb5kdc[10636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047737, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:08:57.310441 11466 init.cc:377] Logged in from keytab as kudu/127.31.131.67@KRBTEST.COM (short username kudu)
I20240507 02:08:57.310868 11466 webserver.cc:460] Webserver started at http://127.31.131.67:39647/ using document root <none> and password file <none>
I20240507 02:08:57.311033 11466 fs_manager.cc:359] Metadata directory not provided
I20240507 02:08:57.311067 11466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:08:57.311158 11466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
May 07 02:08:57 dist-test-slave-v0bg krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047737, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.67@KRBTEST.COM for HTTP/127.31.131.84@KRBTEST.COM
I20240507 02:08:57.327693 11466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/ts-2/data/instance:
uuid: "19d7cbabe9e548b49ff72ff682261a54"
format_stamp: "Formatted at 2024-05-07 02:08:57 on dist-test-slave-v0bg"
server_key: "58f2765d0a094309c711c2bd2fba52f0"
server_key_iv: "ec6f39e8a9abfd0e3e664190ffb665d0"
server_key_version: "kuduclusterkey@0"
I20240507 02:08:57.328115 11466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/ts-2/wal/instance:
uuid: "19d7cbabe9e548b49ff72ff682261a54"
format_stamp: "Formatted at 2024-05-07 02:08:57 on dist-test-slave-v0bg"
server_key: "58f2765d0a094309c711c2bd2fba52f0"
server_key_iv: "ec6f39e8a9abfd0e3e664190ffb665d0"
server_key_version: "kuduclusterkey@0"
I20240507 02:08:57.334968 11466 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.000s
I20240507 02:08:57.350284 11482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:08:57.350430 11466 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.000s
I20240507 02:08:57.350507 11466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/ts-2/wal
uuid: "19d7cbabe9e548b49ff72ff682261a54"
format_stamp: "Formatted at 2024-05-07 02:08:57 on dist-test-slave-v0bg"
server_key: "58f2765d0a094309c711c2bd2fba52f0"
server_key_iv: "ec6f39e8a9abfd0e3e664190ffb665d0"
server_key_version: "kuduclusterkey@0"
I20240507 02:08:57.350607 11466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:08:57.389389 11466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:08:57.392218 11466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:08:57.392338 11466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:08:57.392509 11466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:08:57.392856 11466 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:08:57.392904 11466 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:08:57.392937 11466 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:08:57.392957 11466 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:08:57.399452 11466 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.67:44633
I20240507 02:08:57.399817 11466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
I20240507 02:08:57.400343 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 11466
I20240507 02:08:57.400442 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/ts-2/wal/instance
I20240507 02:08:57.400612 32269 external_mini_cluster.cc:1428] Setting key 72d85c7720236923ed3be897059078da
I20240507 02:08:57.400096 11595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.67:44633 every 8 connection(s)
May 07 02:08:57 dist-test-slave-v0bg krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047737, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.67@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:08:57.409828 11326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:57.400933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.67:60375 (local address 127.31.131.126:35385)
0507 02:08:57.400970 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:57.400973 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:57.401391 (+ 418us) server_negotiation.cc:407] Connection header received
0507 02:08:57.402255 (+ 864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:57.402256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:57.402268 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:57.402305 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:08:57.403153 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:57.403949 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:08:57.404722 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:57.404892 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:57.407093 (+ 2201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:08:57.407096 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:08:57.407097 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:08:57.407103 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:08:57.408589 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:08:57.409037 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:08:57.409038 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:08:57.409038 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:08:57.409061 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:08:57.409237 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:08:57.409237 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:08:57.409238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:08:57.409338 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:08:57.409396 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0507 02:08:57.409656 (+ 260us) server_negotiation.cc:299] Negotiation successful
0507 02:08:57.409738 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20240507 02:08:57.409898 11598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:57.400691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:35385 (local address 127.31.131.67:60375)
0507 02:08:57.401089 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0507 02:08:57.401120 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0507 02:08:57.402132 (+ 1012us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:08:57.402469 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:08:57.402473 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:08:57.402693 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:08:57.403049 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:57.403054 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:57.404054 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:08:57.404055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:08:57.404633 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:57.404635 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:57.404996 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:57.405562 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:08:57.405580 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:08:57.407000 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:08:57.408689 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:08:57.408696 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:08:57.408700 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:08:57.408957 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:08:57.409118 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:08:57.409119 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:08:57.409119 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:08:57.409186 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:08:57.409391 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:08:57.409395 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:08:57.409525 (+ 130us) client_negotiation.cc:769] Sending connection context
0507 02:08:57.409635 (+ 110us) client_negotiation.cc:240] Negotiation successful
0507 02:08:57.409737 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":75,"threads_started":1}
I20240507 02:08:57.410522 11596 heartbeater.cc:347] Connected to a master server at 127.31.131.126:35385
I20240507 02:08:57.410632 11596 heartbeater.cc:464] Registering TS with master...
I20240507 02:08:57.410815 11596 heartbeater.cc:511] Master 127.31.131.126:35385 requested a full tablet report, sending...
I20240507 02:08:57.411180 11112 ts_manager.cc:194] Registered new tserver with Master: 19d7cbabe9e548b49ff72ff682261a54 (127.31.131.67:44633)
I20240507 02:08:57.411640 11112 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.131.67@KRBTEST.COM'} at 127.31.131.67:60375
I20240507 02:08:57.412261 32269 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240507 02:08:57.419890 11326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:57.413560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44522 (local address 127.31.131.126:35385)
0507 02:08:57.413590 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:57.413592 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:57.413832 (+ 240us) server_negotiation.cc:407] Connection header received
0507 02:08:57.413874 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:57.413875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:57.413884 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:57.413939 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:08:57.414807 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:57.415509 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:08:57.416217 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:57.416390 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:57.417431 (+ 1041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:08:57.417434 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:08:57.417435 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:08:57.417440 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:08:57.418980 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:08:57.419307 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:08:57.419308 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:08:57.419308 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:08:57.419328 (+ 20us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:08:57.419460 (+ 132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:08:57.419461 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:08:57.419461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:08:57.419546 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:08:57.419633 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0507 02:08:57.419734 (+ 101us) server_negotiation.cc:299] Negotiation successful
0507 02:08:57.419806 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20240507 02:08:57.421379 11112 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44522:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 02:08:57.423413 11112 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 02:08:57.430598 11610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:57.425956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.65:39857 (local address 127.0.0.1:34262)
0507 02:08:57.426716 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0507 02:08:57.426724 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0507 02:08:57.427274 (+ 550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:08:57.427674 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:08:57.427677 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:08:57.427680 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:08:57.427913 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:57.427915 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:57.429373 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:08:57.429374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:08:57.430353 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:57.430355 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:57.430401 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:57.430431 (+ 30us) client_negotiation.cc:769] Sending connection context
0507 02:08:57.430467 (+ 36us) client_negotiation.cc:240] Negotiation successful
0507 02:08:57.430502 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":745,"thread_start_us":60,"threads_started":1}
I20240507 02:08:57.431140 11612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:57.425735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.66:44837 (local address 127.0.0.1:43318)
0507 02:08:57.426238 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0507 02:08:57.426320 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0507 02:08:57.426602 (+ 282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:08:57.427946 (+ 1344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:08:57.427947 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:08:57.427949 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:08:57.428147 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:57.428149 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:57.429658 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:08:57.429659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:08:57.430975 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:57.430977 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:57.431014 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:57.431017 (+ 3us) client_negotiation.cc:769] Sending connection context
0507 02:08:57.431037 (+ 20us) client_negotiation.cc:240] Negotiation successful
0507 02:08:57.431076 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":68,"threads_started":1}
I20240507 02:08:57.431768 11611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:57.426033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34262 (local address 127.31.131.65:39857)
0507 02:08:57.427485 (+ 1452us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:57.427496 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:57.427507 (+ 11us) server_negotiation.cc:407] Connection header received
0507 02:08:57.427546 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:57.427548 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:57.427565 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:57.427648 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:08:57.428007 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:57.429286 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:08:57.430900 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:57.431572 (+ 672us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:57.431619 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0507 02:08:57.431651 (+ 32us) server_negotiation.cc:299] Negotiation successful
0507 02:08:57.431691 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1421,"thread_start_us":60,"threads_started":1}
I20240507 02:08:57.432060 11609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:57.425801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43318 (local address 127.31.131.66:44837)
0507 02:08:57.427779 (+ 1978us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:57.427786 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:57.427795 (+ 9us) server_negotiation.cc:407] Connection header received
0507 02:08:57.427831 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:57.427832 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:57.427847 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:57.428223 (+ 376us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:08:57.428549 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:57.429565 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:08:57.431390 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:57.431875 (+ 485us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:57.431922 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0507 02:08:57.431952 (+ 30us) server_negotiation.cc:299] Negotiation successful
0507 02:08:57.431989 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1944,"thread_start_us":76,"threads_started":1}
I20240507 02:08:57.432417 11258 tablet_service.cc:1467] Processing CreateTablet for tablet f7d9c6e651ed4adb8a5891d7540dd79e (DEFAULT_TABLE table=test-table [id=b2dd23f9b2dd4458ba5ae06c88ae33dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:08:57.432709 11258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7d9c6e651ed4adb8a5891d7540dd79e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:08:57.435160 11608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:57.426113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.67:44633 (local address 127.0.0.1:48752)
0507 02:08:57.427347 (+ 1234us) negotiation.cc:107] Waiting for socket to connect
0507 02:08:57.427355 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0507 02:08:57.427392 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:08:57.432478 (+ 5086us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:08:57.432479 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:08:57.432481 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:08:57.432684 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:57.432685 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:57.434054 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:08:57.434055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:08:57.434963 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:57.434966 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:57.435036 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:57.435039 (+ 3us) client_negotiation.cc:769] Sending connection context
0507 02:08:57.435067 (+ 28us) client_negotiation.cc:240] Negotiation successful
0507 02:08:57.435099 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1221,"thread_start_us":67,"threads_started":1}
I20240507 02:08:57.435482 11392 tablet_service.cc:1467] Processing CreateTablet for tablet f7d9c6e651ed4adb8a5891d7540dd79e (DEFAULT_TABLE table=test-table [id=b2dd23f9b2dd4458ba5ae06c88ae33dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:08:57.435740 11392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7d9c6e651ed4adb8a5891d7540dd79e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:08:57.435930 11613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:57.426181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48752 (local address 127.31.131.67:44633)
0507 02:08:57.432316 (+ 6135us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:57.432322 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:57.432332 (+ 10us) server_negotiation.cc:407] Connection header received
0507 02:08:57.432361 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:57.432362 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:57.432376 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:57.432461 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:08:57.432819 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:57.433977 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:08:57.435068 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:57.435763 (+ 695us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:57.435802 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0507 02:08:57.435831 (+ 29us) server_negotiation.cc:299] Negotiation successful
0507 02:08:57.435865 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6108,"thread_start_us":64,"threads_started":1}
I20240507 02:08:57.436228 11614 tablet_bootstrap.cc:492] T f7d9c6e651ed4adb8a5891d7540dd79e P 83d42e907cec44108f095074562ae6ac: Bootstrap starting.
I20240507 02:08:57.436398 11530 tablet_service.cc:1467] Processing CreateTablet for tablet f7d9c6e651ed4adb8a5891d7540dd79e (DEFAULT_TABLE table=test-table [id=b2dd23f9b2dd4458ba5ae06c88ae33dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:08:57.436616 11530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7d9c6e651ed4adb8a5891d7540dd79e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:08:57.436741 11614 tablet_bootstrap.cc:654] T f7d9c6e651ed4adb8a5891d7540dd79e P 83d42e907cec44108f095074562ae6ac: Neither blocks nor log segments found. Creating new log.
I20240507 02:08:57.437245 11614 log.cc:824] T f7d9c6e651ed4adb8a5891d7540dd79e P 83d42e907cec44108f095074562ae6ac: Log is configured to *not* fsync() on all Append() calls
I20240507 02:08:57.437837 11614 tablet_bootstrap.cc:492] T f7d9c6e651ed4adb8a5891d7540dd79e P 83d42e907cec44108f095074562ae6ac: No bootstrap required, opened a new log
I20240507 02:08:57.437886 11614 ts_tablet_manager.cc:1367] T f7d9c6e651ed4adb8a5891d7540dd79e P 83d42e907cec44108f095074562ae6ac: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240507 02:08:57.439709 11614 raft_consensus.cc:348] T f7d9c6e651ed4adb8a5891d7540dd79e P 83d42e907cec44108f095074562ae6ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d968304608064d47a46e2d6da3b33106" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44837 } } peers { permanent_uuid: "83d42e907cec44108f095074562ae6ac" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 39857 } } peers { permanent_uuid: "19d7cbabe9e548b49ff72ff682261a54" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 44633 } }
I20240507 02:08:57.439854 11614 raft_consensus.cc:374] T f7d9c6e651ed4adb8a5891d7540dd79e P 83d42e907cec44108f095074562ae6ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:08:57.439878 11614 raft_consensus.cc:729] T f7d9c6e651ed4adb8a5891d7540dd79e P 83d42e907cec44108f095074562ae6ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83d42e907cec44108f095074562ae6ac, State: Initialized, Role: FOLLOWER
I20240507 02:08:57.439968 11614 consensus_queue.cc:260] T f7d9c6e651ed4adb8a5891d7540dd79e P 83d42e907cec44108f095074562ae6ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d968304608064d47a46e2d6da3b33106" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44837 } } peers { permanent_uuid: "83d42e907cec44108f095074562ae6ac" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 39857 } } peers { permanent_uuid: "19d7cbabe9e548b49ff72ff682261a54" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 44633 } }
I20240507 02:08:57.440204 11614 ts_tablet_manager.cc:1398] T f7d9c6e651ed4adb8a5891d7540dd79e P 83d42e907cec44108f095074562ae6ac: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240507 02:08:57.440588 11324 heartbeater.cc:503] Master 127.31.131.126:35385 was elected leader, sending a full tablet report...
I20240507 02:08:57.441146 11616 tablet_bootstrap.cc:492] T f7d9c6e651ed4adb8a5891d7540dd79e P d968304608064d47a46e2d6da3b33106: Bootstrap starting.
I20240507 02:08:57.441624 11616 tablet_bootstrap.cc:654] T f7d9c6e651ed4adb8a5891d7540dd79e P d968304608064d47a46e2d6da3b33106: Neither blocks nor log segments found. Creating new log.
I20240507 02:08:57.442063 11616 log.cc:824] T f7d9c6e651ed4adb8a5891d7540dd79e P d968304608064d47a46e2d6da3b33106: Log is configured to *not* fsync() on all Append() calls
I20240507 02:08:57.442332 11617 tablet_bootstrap.cc:492] T f7d9c6e651ed4adb8a5891d7540dd79e P 19d7cbabe9e548b49ff72ff682261a54: Bootstrap starting.
I20240507 02:08:57.442802 11617 tablet_bootstrap.cc:654] T f7d9c6e651ed4adb8a5891d7540dd79e P 19d7cbabe9e548b49ff72ff682261a54: Neither blocks nor log segments found. Creating new log.
I20240507 02:08:57.443220 11617 log.cc:824] T f7d9c6e651ed4adb8a5891d7540dd79e P 19d7cbabe9e548b49ff72ff682261a54: Log is configured to *not* fsync() on all Append() calls
I20240507 02:08:57.443794 11616 tablet_bootstrap.cc:492] T f7d9c6e651ed4adb8a5891d7540dd79e P d968304608064d47a46e2d6da3b33106: No bootstrap required, opened a new log
I20240507 02:08:57.443840 11616 ts_tablet_manager.cc:1367] T f7d9c6e651ed4adb8a5891d7540dd79e P d968304608064d47a46e2d6da3b33106: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240507 02:08:57.445669 11616 raft_consensus.cc:348] T f7d9c6e651ed4adb8a5891d7540dd79e P d968304608064d47a46e2d6da3b33106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d968304608064d47a46e2d6da3b33106" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44837 } } peers { permanent_uuid: "83d42e907cec44108f095074562ae6ac" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 39857 } } peers { permanent_uuid: "19d7cbabe9e548b49ff72ff682261a54" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 44633 } }
I20240507 02:08:57.445780 11616 raft_consensus.cc:374] T f7d9c6e651ed4adb8a5891d7540dd79e P d968304608064d47a46e2d6da3b33106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:08:57.445799 11616 raft_consensus.cc:729] T f7d9c6e651ed4adb8a5891d7540dd79e P d968304608064d47a46e2d6da3b33106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d968304608064d47a46e2d6da3b33106, State: Initialized, Role: FOLLOWER
I20240507 02:08:57.445894 11616 consensus_queue.cc:260] T f7d9c6e651ed4adb8a5891d7540dd79e P d968304608064d47a46e2d6da3b33106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d968304608064d47a46e2d6da3b33106" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44837 } } peers { permanent_uuid: "83d42e907cec44108f095074562ae6ac" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 39857 } } peers { permanent_uuid: "19d7cbabe9e548b49ff72ff682261a54" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 44633 } }
I20240507 02:08:57.446092 11616 ts_tablet_manager.cc:1398] T f7d9c6e651ed4adb8a5891d7540dd79e P d968304608064d47a46e2d6da3b33106: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240507 02:08:57.446174 11461 heartbeater.cc:503] Master 127.31.131.126:35385 was elected leader, sending a full tablet report...
I20240507 02:08:57.447099 11617 tablet_bootstrap.cc:492] T f7d9c6e651ed4adb8a5891d7540dd79e P 19d7cbabe9e548b49ff72ff682261a54: No bootstrap required, opened a new log
I20240507 02:08:57.447146 11617 ts_tablet_manager.cc:1367] T f7d9c6e651ed4adb8a5891d7540dd79e P 19d7cbabe9e548b49ff72ff682261a54: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
W20240507 02:08:57.448310 11325 tablet.cc:2369] T f7d9c6e651ed4adb8a5891d7540dd79e P 83d42e907cec44108f095074562ae6ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240507 02:08:57.448660 11617 raft_consensus.cc:348] T f7d9c6e651ed4adb8a5891d7540dd79e P 19d7cbabe9e548b49ff72ff682261a54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d968304608064d47a46e2d6da3b33106" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44837 } } peers { permanent_uuid: "83d42e907cec44108f095074562ae6ac" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 39857 } } peers { permanent_uuid: "19d7cbabe9e548b49ff72ff682261a54" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 44633 } }
I20240507 02:08:57.448767 11617 raft_consensus.cc:374] T f7d9c6e651ed4adb8a5891d7540dd79e P 19d7cbabe9e548b49ff72ff682261a54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:08:57.448787 11617 raft_consensus.cc:729] T f7d9c6e651ed4adb8a5891d7540dd79e P 19d7cbabe9e548b49ff72ff682261a54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19d7cbabe9e548b49ff72ff682261a54, State: Initialized, Role: FOLLOWER
I20240507 02:08:57.448864 11617 consensus_queue.cc:260] T f7d9c6e651ed4adb8a5891d7540dd79e P 19d7cbabe9e548b49ff72ff682261a54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d968304608064d47a46e2d6da3b33106" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44837 } } peers { permanent_uuid: "83d42e907cec44108f095074562ae6ac" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 39857 } } peers { permanent_uuid: "19d7cbabe9e548b49ff72ff682261a54" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 44633 } }
I20240507 02:08:57.449048 11617 ts_tablet_manager.cc:1398] T f7d9c6e651ed4adb8a5891d7540dd79e P 19d7cbabe9e548b49ff72ff682261a54: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240507 02:08:57.449133 11596 heartbeater.cc:503] Master 127.31.131.126:35385 was elected leader, sending a full tablet report...
May 07 02:08:57 dist-test-slave-v0bg krb5kdc[10636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047737, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.126@KRBTEST.COM for HTTP/127.31.131.84@KRBTEST.COM
I20240507 02:08:57.650508 11622 raft_consensus.cc:482] T f7d9c6e651ed4adb8a5891d7540dd79e P 19d7cbabe9e548b49ff72ff682261a54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 02:08:57.650671 11622 raft_consensus.cc:504] T f7d9c6e651ed4adb8a5891d7540dd79e P 19d7cbabe9e548b49ff72ff682261a54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d968304608064d47a46e2d6da3b33106" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44837 } } peers { permanent_uuid: "83d42e907cec44108f095074562ae6ac" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 39857 } } peers { permanent_uuid: "19d7cbabe9e548b49ff72ff682261a54" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 44633 } }
I20240507 02:08:57.651032 11622 leader_election.cc:288] T f7d9c6e651ed4adb8a5891d7540dd79e P 19d7cbabe9e548b49ff72ff682261a54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d968304608064d47a46e2d6da3b33106 (127.31.131.66:44837), 83d42e907cec44108f095074562ae6ac (127.31.131.65:39857)
W20240507 02:08:57.650681 11597 tablet.cc:2369] T f7d9c6e651ed4adb8a5891d7540dd79e P 19d7cbabe9e548b49ff72ff682261a54: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240507 02:08:57.654315 11598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:57.651244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.66:44837 (local address 127.31.131.67:58855)
0507 02:08:57.651640 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0507 02:08:57.651651 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0507 02:08:57.651692 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:08:57.652036 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:08:57.652037 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:08:57.652039 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:08:57.652271 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:57.652273 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:57.653191 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:08:57.653192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:08:57.654079 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:57.654081 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:57.654172 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:57.654177 (+ 5us) client_negotiation.cc:769] Sending connection context
0507 02:08:57.654207 (+ 30us) client_negotiation.cc:240] Negotiation successful
0507 02:08:57.654248 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374}
I20240507 02:08:57.655082 11609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:57.651800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.67:58855 (local address 127.31.131.66:44837)
0507 02:08:57.651857 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:57.651859 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:57.651870 (+ 11us) server_negotiation.cc:407] Connection header received
0507 02:08:57.651897 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:57.651898 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:57.651908 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:57.651969 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:08:57.652355 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:57.653096 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:08:57.654476 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:57.654956 (+ 480us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:57.654964 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0507 02:08:57.654988 (+ 24us) server_negotiation.cc:299] Negotiation successful
0507 02:08:57.655018 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20240507 02:08:57.655347 11415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7d9c6e651ed4adb8a5891d7540dd79e" candidate_uuid: "19d7cbabe9e548b49ff72ff682261a54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d968304608064d47a46e2d6da3b33106" is_pre_election: true
I20240507 02:08:57.655520 11415 raft_consensus.cc:2454] T f7d9c6e651ed4adb8a5891d7540dd79e P d968304608064d47a46e2d6da3b33106 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19d7cbabe9e548b49ff72ff682261a54 in term 0.
I20240507 02:08:57.655745 11484 leader_election.cc:302] T f7d9c6e651ed4adb8a5891d7540dd79e P 19d7cbabe9e548b49ff72ff682261a54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19d7cbabe9e548b49ff72ff682261a54, d968304608064d47a46e2d6da3b33106; no voters:
I20240507 02:08:57.655876 11622 raft_consensus.cc:2781] T f7d9c6e651ed4adb8a5891d7540dd79e P 19d7cbabe9e548b49ff72ff682261a54 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 02:08:57.655925 11622 raft_consensus.cc:482] T f7d9c6e651ed4adb8a5891d7540dd79e P 19d7cbabe9e548b49ff72ff682261a54 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 02:08:57.655944 11622 raft_consensus.cc:3037] T f7d9c6e651ed4adb8a5891d7540dd79e P 19d7cbabe9e548b49ff72ff682261a54 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:08:57.656194 11611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:57.651866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.67:41775 (local address 127.31.131.65:39857)
0507 02:08:57.651942 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:57.651944 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:57.652587 (+ 643us) server_negotiation.cc:407] Connection header received
0507 02:08:57.652675 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:57.652676 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:57.652684 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:57.652708 (+ 24us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:08:57.653116 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:57.653821 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:08:57.655270 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:57.655712 (+ 442us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:57.655720 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0507 02:08:57.656092 (+ 372us) server_negotiation.cc:299] Negotiation successful
0507 02:08:57.656133 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20240507 02:08:57.656504 11623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:57.651360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.65:39857 (local address 127.31.131.67:41775)
0507 02:08:57.652526 (+ 1166us) negotiation.cc:107] Waiting for socket to connect
0507 02:08:57.652536 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0507 02:08:57.652611 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:08:57.652766 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:08:57.652767 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:08:57.652768 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:08:57.653037 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:57.653038 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:57.653932 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:08:57.653933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:08:57.655052 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:08:57.655055 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:08:57.656009 (+ 954us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:57.656014 (+ 5us) client_negotiation.cc:769] Sending connection context
0507 02:08:57.656395 (+ 381us) client_negotiation.cc:240] Negotiation successful
0507 02:08:57.656434 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1150,"thread_start_us":90,"threads_started":1}
I20240507 02:08:57.656826 11622 raft_consensus.cc:504] T f7d9c6e651ed4adb8a5891d7540dd79e P 19d7cbabe9e548b49ff72ff682261a54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d968304608064d47a46e2d6da3b33106" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44837 } } peers { permanent_uuid: "83d42e907cec44108f095074562ae6ac" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 39857 } } peers { permanent_uuid: "19d7cbabe9e548b49ff72ff682261a54" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 44633 } }
I20240507 02:08:57.656860 11278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7d9c6e651ed4adb8a5891d7540dd79e" candidate_uuid: "19d7cbabe9e548b49ff72ff682261a54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83d42e907cec44108f095074562ae6ac" is_pre_election: true
I20240507 02:08:57.656946 11622 leader_election.cc:288] T f7d9c6e651ed4adb8a5891d7540dd79e P 19d7cbabe9e548b49ff72ff682261a54 [CANDIDATE]: Term 1 election: Requested vote from peers d968304608064d47a46e2d6da3b33106 (127.31.131.66:44837), 83d42e907cec44108f095074562ae6ac (127.31.131.65:39857)
I20240507 02:08:57.656976 11278 raft_consensus.cc:2454] T f7d9c6e651ed4adb8a5891d7540dd79e P 83d42e907cec44108f095074562ae6ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19d7cbabe9e548b49ff72ff682261a54 in term 0.
W20240507 02:08:57.657053 11462 tablet.cc:2369] T f7d9c6e651ed4adb8a5891d7540dd79e P d968304608064d47a46e2d6da3b33106: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240507 02:08:57.657344 11278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7d9c6e651ed4adb8a5891d7540dd79e" candidate_uuid: "19d7cbabe9e548b49ff72ff682261a54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83d42e907cec44108f095074562ae6ac"
I20240507 02:08:57.657361 11415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7d9c6e651ed4adb8a5891d7540dd79e" candidate_uuid: "19d7cbabe9e548b49ff72ff682261a54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d968304608064d47a46e2d6da3b33106"
I20240507 02:08:57.657406 11278 raft_consensus.cc:3037] T f7d9c6e651ed4adb8a5891d7540dd79e P 83d42e907cec44108f095074562ae6ac [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:08:57.657428 11415 raft_consensus.cc:3037] T f7d9c6e651ed4adb8a5891d7540dd79e P d968304608064d47a46e2d6da3b33106 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:08:57.658201 11278 raft_consensus.cc:2454] T f7d9c6e651ed4adb8a5891d7540dd79e P 83d42e907cec44108f095074562ae6ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19d7cbabe9e548b49ff72ff682261a54 in term 1.
I20240507 02:08:57.658236 11415 raft_consensus.cc:2454] T f7d9c6e651ed4adb8a5891d7540dd79e P d968304608064d47a46e2d6da3b33106 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19d7cbabe9e548b49ff72ff682261a54 in term 1.
I20240507 02:08:57.658383 11484 leader_election.cc:302] T f7d9c6e651ed4adb8a5891d7540dd79e P 19d7cbabe9e548b49ff72ff682261a54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19d7cbabe9e548b49ff72ff682261a54, d968304608064d47a46e2d6da3b33106; no voters:
I20240507 02:08:57.658457 11622 raft_consensus.cc:2781] T f7d9c6e651ed4adb8a5891d7540dd79e P 19d7cbabe9e548b49ff72ff682261a54 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:08:57.658587 11622 raft_consensus.cc:686] T f7d9c6e651ed4adb8a5891d7540dd79e P 19d7cbabe9e548b49ff72ff682261a54 [term 1 LEADER]: Becoming Leader. State: Replica: 19d7cbabe9e548b49ff72ff682261a54, State: Running, Role: LEADER
I20240507 02:08:57.658676 11622 consensus_queue.cc:237] T f7d9c6e651ed4adb8a5891d7540dd79e P 19d7cbabe9e548b49ff72ff682261a54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d968304608064d47a46e2d6da3b33106" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44837 } } peers { permanent_uuid: "83d42e907cec44108f095074562ae6ac" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 39857 } } peers { permanent_uuid: "19d7cbabe9e548b49ff72ff682261a54" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 44633 } }
I20240507 02:08:57.659323 11112 catalog_manager.cc:5484] T f7d9c6e651ed4adb8a5891d7540dd79e P 19d7cbabe9e548b49ff72ff682261a54 reported cstate change: term changed from 0 to 1, leader changed from <none> to 19d7cbabe9e548b49ff72ff682261a54 (127.31.131.67). New cstate: current_term: 1 leader_uuid: "19d7cbabe9e548b49ff72ff682261a54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d968304608064d47a46e2d6da3b33106" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83d42e907cec44108f095074562ae6ac" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 39857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19d7cbabe9e548b49ff72ff682261a54" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 44633 } health_report { overall_health: HEALTHY } } }
I20240507 02:08:57.712478 11613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:57.709661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48754 (local address 127.31.131.67:44633)
0507 02:08:57.709705 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0507 02:08:57.709707 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:08:57.709714 (+ 7us) server_negotiation.cc:407] Connection header received
0507 02:08:57.709786 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:08:57.709786 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:08:57.709804 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:08:57.709869 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0507 02:08:57.710205 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:57.710940 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:08:57.711806 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:08:57.711977 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:08:57.712016 (+ 39us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0507 02:08:57.712125 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0507 02:08:57.712195 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0507 02:08:57.712367 (+ 172us) server_negotiation.cc:299] Negotiation successful
0507 02:08:57.712390 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20240507 02:08:57.714232 11278 raft_consensus.cc:1261] T f7d9c6e651ed4adb8a5891d7540dd79e P 83d42e907cec44108f095074562ae6ac [term 1 FOLLOWER]: Refusing update from remote peer 19d7cbabe9e548b49ff72ff682261a54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 02:08:57.714232 11415 raft_consensus.cc:1261] T f7d9c6e651ed4adb8a5891d7540dd79e P d968304608064d47a46e2d6da3b33106 [term 1 FOLLOWER]: Refusing update from remote peer 19d7cbabe9e548b49ff72ff682261a54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 02:08:57.714578 11624 consensus_queue.cc:1035] T f7d9c6e651ed4adb8a5891d7540dd79e P 19d7cbabe9e548b49ff72ff682261a54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83d42e907cec44108f095074562ae6ac" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 39857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 02:08:57.714687 11622 consensus_queue.cc:1035] T f7d9c6e651ed4adb8a5891d7540dd79e P 19d7cbabe9e548b49ff72ff682261a54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d968304608064d47a46e2d6da3b33106" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 02:08:57.717307 11627 mvcc.cc:205] Tried to move back new op lower bound from 7024835533675151360 to 7024835533450375168. Current Snapshot: MvccSnapshot[applied={T|T < 7024835533675151360}]
I20240507 02:08:57.718158 11628 mvcc.cc:205] Tried to move back new op lower bound from 7024835533675151360 to 7024835533450375168. Current Snapshot: MvccSnapshot[applied={T|T < 7024835533675151360}]
I20240507 02:08:57.719919 11629 mvcc.cc:205] Tried to move back new op lower bound from 7024835533675151360 to 7024835533450375168. Current Snapshot: MvccSnapshot[applied={T|T < 7024835533675151360}]
I20240507 02:08:57.727154 11111 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44522:
table { table_name: "test-table" } modify_external_catalogs: true
I20240507 02:08:57.727246 11111 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44522:
table { table_name: "test-table" } modify_external_catalogs: true
I20240507 02:08:57.727896 11111 catalog_manager.cc:5771] T 00000000000000000000000000000000 P d6d81953e1e74463b26017e31a910fa9: Sending DeleteTablet for 3 replicas of tablet f7d9c6e651ed4adb8a5891d7540dd79e
I20240507 02:08:57.728286 11530 tablet_service.cc:1514] Processing DeleteTablet for tablet f7d9c6e651ed4adb8a5891d7540dd79e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:08:57 UTC) from {username='kudu', principal='kudu/127.31.131.126@KRBTEST.COM'} at 127.0.0.1:48752
I20240507 02:08:57.728329 11392 tablet_service.cc:1514] Processing DeleteTablet for tablet f7d9c6e651ed4adb8a5891d7540dd79e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:08:57 UTC) from {username='kudu', principal='kudu/127.31.131.126@KRBTEST.COM'} at 127.0.0.1:43318
I20240507 02:08:57.728507 11258 tablet_service.cc:1514] Processing DeleteTablet for tablet f7d9c6e651ed4adb8a5891d7540dd79e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:08:57 UTC) from {username='kudu', principal='kudu/127.31.131.126@KRBTEST.COM'} at 127.0.0.1:34262
I20240507 02:08:57.728996 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 11193
I20240507 02:08:57.738633 11638 tablet_replica.cc:330] T f7d9c6e651ed4adb8a5891d7540dd79e P d968304608064d47a46e2d6da3b33106: stopping tablet replica
I20240507 02:08:57.738725 11638 raft_consensus.cc:2229] T f7d9c6e651ed4adb8a5891d7540dd79e P d968304608064d47a46e2d6da3b33106 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 02:08:57.738826 11638 raft_consensus.cc:2258] T f7d9c6e651ed4adb8a5891d7540dd79e P d968304608064d47a46e2d6da3b33106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 02:08:57.742628 11637 tablet_replica.cc:330] T f7d9c6e651ed4adb8a5891d7540dd79e P 19d7cbabe9e548b49ff72ff682261a54: stopping tablet replica
I20240507 02:08:57.742668 11638 ts_tablet_manager.cc:1875] T f7d9c6e651ed4adb8a5891d7540dd79e P d968304608064d47a46e2d6da3b33106: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 02:08:57.742713 11637 raft_consensus.cc:2229] T f7d9c6e651ed4adb8a5891d7540dd79e P 19d7cbabe9e548b49ff72ff682261a54 [term 1 LEADER]: Raft consensus shutting down.
I20240507 02:08:57.742810 11637 raft_consensus.cc:2258] T f7d9c6e651ed4adb8a5891d7540dd79e P 19d7cbabe9e548b49ff72ff682261a54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 02:08:57.743198 11637 ts_tablet_manager.cc:1875] T f7d9c6e651ed4adb8a5891d7540dd79e P 19d7cbabe9e548b49ff72ff682261a54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 02:08:57.744192 11638 ts_tablet_manager.cc:1888] T f7d9c6e651ed4adb8a5891d7540dd79e P d968304608064d47a46e2d6da3b33106: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240507 02:08:57.744261 11638 log.cc:1196] T f7d9c6e651ed4adb8a5891d7540dd79e P d968304608064d47a46e2d6da3b33106: Deleting WAL directory at /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/ts-1/wal/wals/f7d9c6e651ed4adb8a5891d7540dd79e
I20240507 02:08:57.744513 11638 ts_tablet_manager.cc:1909] T f7d9c6e651ed4adb8a5891d7540dd79e P d968304608064d47a46e2d6da3b33106: Deleting consensus metadata
I20240507 02:08:57.744632 11637 ts_tablet_manager.cc:1888] T f7d9c6e651ed4adb8a5891d7540dd79e P 19d7cbabe9e548b49ff72ff682261a54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240507 02:08:57.744693 11637 log.cc:1196] T f7d9c6e651ed4adb8a5891d7540dd79e P 19d7cbabe9e548b49ff72ff682261a54: Deleting WAL directory at /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1715047604443657-32269-0/minicluster-data/ts-2/wal/wals/f7d9c6e651ed4adb8a5891d7540dd79e
I20240507 02:08:57.744881 11097 catalog_manager.cc:4830] TS d968304608064d47a46e2d6da3b33106 (127.31.131.66:44837): tablet f7d9c6e651ed4adb8a5891d7540dd79e (table test-table [id=b2dd23f9b2dd4458ba5ae06c88ae33dd]) successfully deleted
I20240507 02:08:57.744920 11637 ts_tablet_manager.cc:1909] T f7d9c6e651ed4adb8a5891d7540dd79e P 19d7cbabe9e548b49ff72ff682261a54: Deleting consensus metadata
W20240507 02:08:57.745070 11098 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.131.65:39857 (error 108)
I20240507 02:08:57.745321 11096 catalog_manager.cc:4830] TS 19d7cbabe9e548b49ff72ff682261a54 (127.31.131.67:44633): tablet f7d9c6e651ed4adb8a5891d7540dd79e (table test-table [id=b2dd23f9b2dd4458ba5ae06c88ae33dd]) successfully deleted
I20240507 02:08:57.754460 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 11330
I20240507 02:08:57.754992 11608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:08:57.754613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.65:39857 (local address 127.0.0.1:34276)
0507 02:08:57.754921 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0507 02:08:57.754961 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.65:39857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287}
W20240507 02:08:57.755146 11098 catalog_manager.cc:4573] TS 83d42e907cec44108f095074562ae6ac (127.31.131.65:39857): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f7d9c6e651ed4adb8a5891d7540dd79e: Network error: Client connection negotiation failed: client connection to 127.31.131.65:39857: connect: Connection refused (error 111)
I20240507 02:08:57.762269 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 11466
I20240507 02:08:57.768184 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 11080
I20240507 02:08:57.774257 32269 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20240507 02:08:58.107754 32269 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20240507 02:08:58.107844 32269 mini_ranger.cc:66] Stopping Ranger...
I20240507 02:08:58.441380 32269 mini_ranger.cc:68] Stopped Ranger
2024-05-07 02:08:58.441 UTC [10666] LOG: received smart shutdown request
2024-05-07 02:08:58.444 UTC [10666] LOG: background worker "logical replication launcher" (PID 10675) exited with exit code 1
2024-05-07 02:08:58.444 UTC [10670] LOG: shutting down
2024-05-07 02:08:58.603 UTC [10666] LOG: database system is shut down
2024-05-07T02:08:58Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (40642 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 07 02:08:58 dist-test-slave-v0bg krb5kdc[11649](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 07 02:08:58 dist-test-slave-v0bg krb5kdc[11649](info): set up 2 sockets
May 07 02:08:58 dist-test-slave-v0bg krb5kdc[11649](info): commencing operation
krb5kdc: starting...
W20240507 02:08:59.583118 32269 mini_kdc.cc:121] Time spent starting KDC: real 0.867s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/krb5kdc/test-user.keytab.
May 07 02:08:59 dist-test-slave-v0bg krb5kdc[11649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-07T02:08:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-07T02:08:59Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.31.131.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.31.131.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.31.131.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/krb5kdc/rangeradmin_127.31.131.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.31.131.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/krb5kdc/rangeradmin_127.31.131.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.31.131.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.31.131.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.31.131.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/krb5kdc/rangerlookup_127.31.131.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.31.131.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/krb5kdc/rangerlookup_127.31.131.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.31.131.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.31.131.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.31.131.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/krb5kdc/HTTP_127.31.131.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.31.131.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/krb5kdc/HTTP_127.31.131.84@KRBTEST.COM.keytab.
I20240507 02:08:59.710767 32269 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskTztKno/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/postgres -l logfile start
2024-05-07 02:09:01.871 UTC [11679] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
2024-05-07 02:09:01.871 UTC [11679] LOG: listening on IPv4 address "127.31.131.84", port 41901
2024-05-07 02:09:01.873 UTC [11679] LOG: listening on Unix socket "/tmp/.s.PGSQL.41901"
2024-05-07 02:09:01.891 UTC [11682] LOG: database system was shut down at 2024-05-07 02:09:00 UTC
2024-05-07 02:09:01.894 UTC [11679] LOG: database system is ready to accept connections
I20240507 02:09:02.703042 32269 mini_postgres.cc:96] Postgres bound to 41901
2024-05-07 02:09:02.707 UTC [11690] FATAL: database "slave" does not exist
127.31.131.84:41901 - accepting connections
I20240507 02:09:02.708181 32269 mini_ranger.cc:161] Starting Ranger...
I20240507 02:09:02.720594 32269 mini_ranger.cc:84] Created miniranger Postgres user
I20240507 02:09:03.086843 32269 mini_ranger.cc:87] Created ranger Postgres database
I20240507 02:09:03.086905 32269 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ranger-admin
2024-05-07 02:09:03,278 [I] DB FLAVOR :POSTGRES
2024-05-07 02:09:03,278 [I] --------- Verifying Ranger DB connection ---------
2024-05-07 02:09:03,278 [I] Checking connection..
2024-05-07 02:09:03,278 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTztKno/build/release/bin/postgresql.jar:/tmp/dist-test-taskTztKno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.131.84:41901/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2024-05-07 02:09:03,512 [I] Checking connection passed.
2024-05-07 02:09:03,512 [I] --------- Verifying version history table ---------
2024-05-07 02:09:03,512 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTztKno/build/release/bin/postgresql.jar:/tmp/dist-test-taskTztKno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.131.84:41901/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2024-05-07 02:09:03,742 [I] Table x_db_version_h does not exist in database ranger
2024-05-07 02:09:03,742 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTztKno/build/release/bin/postgresql.jar:/tmp/dist-test-taskTztKno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.131.84:41901/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2024-05-07 02:09:03,964 [I] Table x_db_version_h does not exist in database ranger
2024-05-07 02:09:03,965 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2024-05-07 02:09:03,965 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTztKno/build/release/bin/postgresql.jar:/tmp/dist-test-taskTztKno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.131.84:41901/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskTztKno/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2024-05-07 02:09:04.202 UTC [11754] WARNING: there is no transaction in progress
2024-05-07 02:09:04,206 [I] create_dbversion_catalog.sql file imported successfully
2024-05-07 02:09:04,206 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTztKno/build/release/bin/postgresql.jar:/tmp/dist-test-taskTztKno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.131.84:41901/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2024-05-07 02:09:04,452 [I] Table x_db_version_h already exists in database 'ranger'
2024-05-07 02:09:04,452 [I] --------- Importing Ranger Core DB Schema ---------
2024-05-07 02:09:04,453 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTztKno/build/release/bin/postgresql.jar:/tmp/dist-test-taskTztKno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.131.84:41901/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2024-05-07 02:09:04,686 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTztKno/build/release/bin/postgresql.jar:/tmp/dist-test-taskTztKno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.131.84:41901/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2024-05-07 02:09:04,923 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTztKno/build/release/bin/postgresql.jar:/tmp/dist-test-taskTztKno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.131.84:41901/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-v0bg.c.gcp-upstream.internal','N') ;"
2024-05-07 02:09:05,149 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTztKno/build/release/bin/postgresql.jar:/tmp/dist-test-taskTztKno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.131.84:41901/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2024-05-07 02:09:05,384 [I] Table x_portal_user does not exist in database ranger
2024-05-07 02:09:05,384 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTztKno/build/release/bin/postgresql.jar:/tmp/dist-test-taskTztKno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.131.84:41901/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2024-05-07 02:09:05,617 [I] Table x_policy_ref_group does not exist in database ranger
2024-05-07 02:09:05,617 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTztKno/build/release/bin/postgresql.jar:/tmp/dist-test-taskTztKno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.131.84:41901/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2024-05-07 02:09:05,832 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTztKno/build/release/bin/postgresql.jar:/tmp/dist-test-taskTztKno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.131.84:41901/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2024-05-07 02:09:06,069 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2024-05-07 02:09:06,069 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTztKno/build/release/bin/postgresql.jar:/tmp/dist-test-taskTztKno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.131.84:41901/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskTztKno/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2024-05-07 02:09:06.890 UTC [11862] WARNING: there is no transaction in progress
2024-05-07 02:09:06.904 UTC [11862] WARNING: there is no transaction in progress
2024-05-07 02:09:06.918 UTC [11862] WARNING: there is no transaction in progress
2024-05-07 02:09:06.931 UTC [11862] WARNING: there is no transaction in progress
2024-05-07 02:09:06.944 UTC [11862] WARNING: there is no transaction in progress
2024-05-07 02:09:06.960 UTC [11862] WARNING: there is no transaction in progress
2024-05-07 02:09:07.007 UTC [11862] WARNING: there is no transaction in progress
2024-05-07 02:09:07.013 UTC [11862] WARNING: there is no transaction in progress
2024-05-07 02:09:07.025 UTC [11862] WARNING: there is no transaction in progress
2024-05-07 02:09:07.034 UTC [11862] WARNING: there is no transaction in progress
2024-05-07 02:09:07.043 UTC [11862] WARNING: there is no transaction in progress
2024-05-07 02:09:07.054 UTC [11862] WARNING: there is no transaction in progress
2024-05-07 02:09:07.063 UTC [11862] WARNING: there is no transaction in progress
2024-05-07 02:09:07.068 UTC [11862] WARNING: there is no transaction in progress
2024-05-07 02:09:07.074 UTC [11862] WARNING: there is no transaction in progress
2024-05-07 02:09:07.079 UTC [11862] WARNING: there is no transaction in progress
2024-05-07 02:09:07,609 [I] ranger_core_db_postgres.sql file imported successfully
2024-05-07 02:09:07,609 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTztKno/build/release/bin/postgresql.jar:/tmp/dist-test-taskTztKno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.131.84:41901/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2024-05-07 02:09:07,816 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2024-05-07 02:09:07,817 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTztKno/build/release/bin/postgresql.jar:/tmp/dist-test-taskTztKno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.131.84:41901/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2024-05-07 02:09:08,036 [I] Table x_portal_user already exists in database 'ranger'
2024-05-07 02:09:08,036 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTztKno/build/release/bin/postgresql.jar:/tmp/dist-test-taskTztKno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.131.84:41901/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2024-05-07 02:09:08,270 [I] Table x_policy_ref_group already exists in database 'ranger'
2024-05-07 02:09:08,270 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTztKno/build/release/bin/postgresql.jar:/tmp/dist-test-taskTztKno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.131.84:41901/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2024-05-07 02:09:08,494 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTztKno/build/release/bin/postgresql.jar:/tmp/dist-test-taskTztKno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.131.84:41901/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2024-05-07 02:09:08,726 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTztKno/build/release/bin/postgresql.jar:/tmp/dist-test-taskTztKno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.131.84:41901/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-v0bg.c.gcp-upstream.internal';"
2024-05-07 02:09:08,943 [I] CORE_DB_SCHEMA import status has been updated
2024-05-07 02:09:08,943 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTztKno/build/release/bin/postgresql.jar:/tmp/dist-test-taskTztKno/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.131.84:41901/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2024-05-07 02:09:09,177 [I] DB_PATCHES have already been applied
I20240507 02:09:09.182785 32269 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ranger-admin:/tmp/dist-test-taskTztKno/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskTztKno/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskTztKno/build/release/bin/postgresql.jar:/tmp/dist-test-taskTztKno/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20240507 02:09:09.182848 32269 mini_ranger.cc:193] Using host: 127.31.131.84
I20240507 02:09:09.185138 32269 mini_ranger.cc:227] Ranger admin URL: http://127.31.131.84:34841
May 07, 2024 2:09:09 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
May 07, 2024 2:09:09 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
May 07, 2024 2:09:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ranger-admin/ews/webapp
May 07, 2024 2:09:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
May 07, 2024 2:09:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ranger-admin/ews/webapp] .....
May 07, 2024 2:09:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ranger-admin/ews/webapp].
May 07, 2024 2:09:10 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20240507 02:09:10.105901 32269 mini_ranger.cc:160] Time spent starting Ranger: real 7.398s user 0.001s sys 0.006s
May 07, 2024 2:09:29 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
May 07, 2024 2:09:29 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.AssetREST
May 07, 2024 2:09:29 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
May 07, 2024 2:09:29 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
May 07, 2024 2:09:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
May 07, 2024 2:09:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
May 07, 2024 2:09:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
May 07, 2024 2:09:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
May 07, 2024 2:09:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
May 07, 2024 2:09:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
May 07, 2024 2:09:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
May 07, 2024 2:09:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
May 07, 2024 2:09:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
May 07, 2024 2:09:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
May 07, 2024 2:09:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
May 07, 2024 2:09:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
May 07, 2024 2:09:29 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
May 07, 2024 2:09:30 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20240507 02:09:30.611055 32269 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.31.131.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.31.131.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.31.131.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/krb5kdc/rangerkms_127.31.131.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.31.131.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/krb5kdc/rangerkms_127.31.131.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.31.131.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.31.131.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.31.131.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/krb5kdc/HTTP_127.31.131.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.31.131.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/krb5kdc/HTTP_127.31.131.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
May 07 02:09:30 dist-test-slave-v0bg krb5kdc[11649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047770, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20240507 02:09:30.691424 32269 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20240507 02:09:30.699695 32269 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20240507 02:09:31.372196 32269 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20240507 02:09:31.372284 32269 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ranger-kms
I20240507 02:09:31.372299 32269 mini_ranger_kms.cc:227] Using postgres at 127.31.131.84:41901
2024-05-07 02:09:31,510 [I] DB FLAVOR :POSTGRES
2024-05-07 02:09:31,511 [I] --------- Verifying Ranger DB connection ---------
2024-05-07 02:09:31,511 [I] Checking connection
2024-05-07 02:09:31,511 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTztKno/build/release/bin/postgresql.jar:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.131.84:41901/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2024-05-07 02:09:31,738 [I] connection success
2024-05-07 02:09:31,738 [I] --------- Verifying Ranger DB tables ---------
2024-05-07 02:09:31,738 [I] Verifying table ranger_masterkey in database rangerkms
2024-05-07 02:09:31,738 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTztKno/build/release/bin/postgresql.jar:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.131.84:41901/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2024-05-07 02:09:31,971 [I] Table ranger_masterkey does not exist in database rangerkms
2024-05-07 02:09:31,971 [I] --------- Importing Ranger Core DB Schema ---------
2024-05-07 02:09:31,972 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2024-05-07 02:09:31,972 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTztKno/build/release/bin/postgresql.jar:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.131.84:41901/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2024-05-07 02:09:32,215 [I] kms_core_db_postgres.sql DB schema imported successfully
I20240507 02:09:32.632845 32269 mini_ranger_kms.cc:310] Created kms service
I20240507 02:09:32.798975 32269 mini_ranger_kms.cc:326] Created kudu user
I20240507 02:09:32.836639 32269 mini_ranger_kms.cc:343] Created rangerkms user
May 07 02:09:32 dist-test-slave-v0bg krb5kdc[11649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047770, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.31.131.84@KRBTEST.COM
I20240507 02:09:33.181381 32269 mini_ranger_kms.cc:384] Added ranger policy
I20240507 02:09:33.181581 32269 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ranger-kms:/tmp/dist-test-taskTztKno/build/release/bin/postgresql.jar:/tmp/dist-test-taskTztKno/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskTztKno/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20240507 02:09:33.181607 32269 mini_ranger_kms.cc:240] Using host: 127.31.131.84
I20240507 02:09:33.184427 32269 mini_ranger_kms.cc:276] Ranger KMS PID: 12042
I20240507 02:09:33.184504 32269 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.31.131.84:35369
May 07, 2024 2:09:34 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
May 07, 2024 2:09:34 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
May 07, 2024 2:09:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
May 07, 2024 2:09:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ranger-kms/ews/webapp] .....
May 07, 2024 2:09:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ranger-kms/ews/webapp].
May 07, 2024 2:09:34 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20240507 02:09:35.030795 32269 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.339s user 0.002s sys 0.120s
I20240507 02:09:35.030879 32269 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20240507 02:09:35.030921 32269 mini_ranger_kms.cc:401] 127.31.131.84:35369/kms/v1/keys
May 07 02:09:35 dist-test-slave-v0bg krb5kdc[11649](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1715047775, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.131.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
May 07 02:09:35 dist-test-slave-v0bg krb5kdc[11649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047775, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.131.84@KRBTEST.COM for HTTP/127.31.131.84@KRBTEST.COM
May 07 02:09:35 dist-test-slave-v0bg krb5kdc[11649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047775, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.131.84@KRBTEST.COM for HTTP/127.31.131.84@KRBTEST.COM
May 07 02:09:37 dist-test-slave-v0bg krb5kdc[11649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.31.131.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab.
Entry for principal kudu/127.31.131.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab.
I20240507 02:09:37.986320 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:34641
--webserver_interface=127.31.131.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab
--principal=kudu/127.31.131.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:37949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.131.126:34641
--ranger_config_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.131.84:35369/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:09:38.058703 12093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:09:38.058863 12093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:09:38.058885 12093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:09:38.060108 12093 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240507 02:09:38.060144 12093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:09:38.060156 12093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:09:38.060166 12093 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240507 02:09:38.060175 12093 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240507 02:09:38.061636 12093 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:37949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/master-0/wal
--ranger_kms_url=127.31.131.84:35369/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.31.131.126:34641
--ranger_config_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab
--principal=kudu/127.31.131.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:34641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.131.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:09:38.061807 12093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:09:38.061975 12093 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:09:38.065357 12099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:09:38.065357 12101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:09:38.065377 12098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:09:38.065425 12093 server_base.cc:1034] running on GCE node
I20240507 02:09:38.065850 12093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:09:38.066038 12093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:09:38.067168 12093 hybrid_clock.cc:648] HybridClock initialized: now 1715047778067148 us; error 37 us; skew 500 ppm
May 07 02:09:38 dist-test-slave-v0bg krb5kdc[11649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047778, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:09:38.072413 12093 init.cc:377] Logged in from keytab as kudu/127.31.131.126@KRBTEST.COM (short username kudu)
I20240507 02:09:38.072801 12093 webserver.cc:460] Webserver started at http://127.31.131.126:37149/ using document root <none> and password file <none>
I20240507 02:09:38.072968 12093 fs_manager.cc:359] Metadata directory not provided
I20240507 02:09:38.073005 12093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:09:38.073110 12093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
May 07 02:09:38 dist-test-slave-v0bg krb5kdc[11649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047778, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.126@KRBTEST.COM for HTTP/127.31.131.84@KRBTEST.COM
I20240507 02:09:38.128297 12093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/master-0/data/instance:
uuid: "902d0b3c51a44c65b0d2f7e6ffe9a301"
format_stamp: "Formatted at 2024-05-07 02:09:38 on dist-test-slave-v0bg"
server_key: "169afc9e7cc8e673864f33df8fc7462f"
server_key_iv: "2c8bed8796ed2270c2f1f4d2a751d07b"
server_key_version: "kuduclusterkey@0"
I20240507 02:09:38.128580 12093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/master-0/wal/instance:
uuid: "902d0b3c51a44c65b0d2f7e6ffe9a301"
format_stamp: "Formatted at 2024-05-07 02:09:38 on dist-test-slave-v0bg"
server_key: "169afc9e7cc8e673864f33df8fc7462f"
server_key_iv: "2c8bed8796ed2270c2f1f4d2a751d07b"
server_key_version: "kuduclusterkey@0"
I20240507 02:09:38.129601 12093 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240507 02:09:38.141662 12108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:09:38.141831 12093 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240507 02:09:38.141908 12093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/master-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/master-0/wal
uuid: "902d0b3c51a44c65b0d2f7e6ffe9a301"
format_stamp: "Formatted at 2024-05-07 02:09:38 on dist-test-slave-v0bg"
server_key: "169afc9e7cc8e673864f33df8fc7462f"
server_key_iv: "2c8bed8796ed2270c2f1f4d2a751d07b"
server_key_version: "kuduclusterkey@0"
I20240507 02:09:38.141975 12093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:09:38.154142 12093 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:09:38.155849 12093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:09:38.155938 12093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:09:38.159860 12093 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.126:34641
I20240507 02:09:38.159909 12160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.126:34641 every 8 connection(s)
I20240507 02:09:38.160214 12093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
I20240507 02:09:38.160734 12161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:09:38.162621 12161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 902d0b3c51a44c65b0d2f7e6ffe9a301: Bootstrap starting.
I20240507 02:09:38.163089 12161 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 902d0b3c51a44c65b0d2f7e6ffe9a301: Neither blocks nor log segments found. Creating new log.
I20240507 02:09:38.163306 12161 log.cc:824] T 00000000000000000000000000000000 P 902d0b3c51a44c65b0d2f7e6ffe9a301: Log is configured to *not* fsync() on all Append() calls
I20240507 02:09:38.163839 12161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 902d0b3c51a44c65b0d2f7e6ffe9a301: No bootstrap required, opened a new log
I20240507 02:09:38.165486 12161 raft_consensus.cc:348] T 00000000000000000000000000000000 P 902d0b3c51a44c65b0d2f7e6ffe9a301 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "902d0b3c51a44c65b0d2f7e6ffe9a301" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 34641 } }
I20240507 02:09:38.165589 12161 raft_consensus.cc:374] T 00000000000000000000000000000000 P 902d0b3c51a44c65b0d2f7e6ffe9a301 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:09:38.165604 12161 raft_consensus.cc:729] T 00000000000000000000000000000000 P 902d0b3c51a44c65b0d2f7e6ffe9a301 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 902d0b3c51a44c65b0d2f7e6ffe9a301, State: Initialized, Role: FOLLOWER
I20240507 02:09:38.165705 12161 consensus_queue.cc:260] T 00000000000000000000000000000000 P 902d0b3c51a44c65b0d2f7e6ffe9a301 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "902d0b3c51a44c65b0d2f7e6ffe9a301" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 34641 } }
I20240507 02:09:38.165766 12161 raft_consensus.cc:388] T 00000000000000000000000000000000 P 902d0b3c51a44c65b0d2f7e6ffe9a301 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 02:09:38.165784 12161 raft_consensus.cc:482] T 00000000000000000000000000000000 P 902d0b3c51a44c65b0d2f7e6ffe9a301 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 02:09:38.165822 12161 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 902d0b3c51a44c65b0d2f7e6ffe9a301 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:09:38.166255 12161 raft_consensus.cc:504] T 00000000000000000000000000000000 P 902d0b3c51a44c65b0d2f7e6ffe9a301 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "902d0b3c51a44c65b0d2f7e6ffe9a301" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 34641 } }
I20240507 02:09:38.166332 12161 leader_election.cc:302] T 00000000000000000000000000000000 P 902d0b3c51a44c65b0d2f7e6ffe9a301 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 902d0b3c51a44c65b0d2f7e6ffe9a301; no voters:
I20240507 02:09:38.166460 12161 leader_election.cc:288] T 00000000000000000000000000000000 P 902d0b3c51a44c65b0d2f7e6ffe9a301 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240507 02:09:38.166505 12164 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 902d0b3c51a44c65b0d2f7e6ffe9a301 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:09:38.166671 12164 raft_consensus.cc:686] T 00000000000000000000000000000000 P 902d0b3c51a44c65b0d2f7e6ffe9a301 [term 1 LEADER]: Becoming Leader. State: Replica: 902d0b3c51a44c65b0d2f7e6ffe9a301, State: Running, Role: LEADER
I20240507 02:09:38.166707 12161 sys_catalog.cc:564] T 00000000000000000000000000000000 P 902d0b3c51a44c65b0d2f7e6ffe9a301 [sys.catalog]: configured and running, proceeding with master startup.
I20240507 02:09:38.167842 12164 consensus_queue.cc:237] T 00000000000000000000000000000000 P 902d0b3c51a44c65b0d2f7e6ffe9a301 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "902d0b3c51a44c65b0d2f7e6ffe9a301" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 34641 } }
I20240507 02:09:38.168339 12165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 902d0b3c51a44c65b0d2f7e6ffe9a301 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 902d0b3c51a44c65b0d2f7e6ffe9a301. Latest consensus state: current_term: 1 leader_uuid: "902d0b3c51a44c65b0d2f7e6ffe9a301" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "902d0b3c51a44c65b0d2f7e6ffe9a301" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 34641 } } }
I20240507 02:09:38.168414 12161 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20240507 02:09:38.168490 12166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 902d0b3c51a44c65b0d2f7e6ffe9a301 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "902d0b3c51a44c65b0d2f7e6ffe9a301" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "902d0b3c51a44c65b0d2f7e6ffe9a301" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 34641 } } }
I20240507 02:09:38.168529 12166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 902d0b3c51a44c65b0d2f7e6ffe9a301 [sys.catalog]: This master's current role is: LEADER
I20240507 02:09:38.168417 12165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 902d0b3c51a44c65b0d2f7e6ffe9a301 [sys.catalog]: This master's current role is: LEADER
I20240507 02:09:38.170123 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 12093
I20240507 02:09:38.170241 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/master-0/wal/instance
I20240507 02:09:38.170440 32269 external_mini_cluster.cc:1428] Setting key 3cb0d6b456e2cc59ac6519f5a5ed6c05
I20240507 02:09:38.171108 12178 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 02:09:38.172039 12178 catalog_manager.cc:1484] Initializing Kudu cluster ID...
May 07 02:09:38 dist-test-slave-v0bg krb5kdc[11649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:09:38.179992 12178 catalog_manager.cc:1347] Generated new cluster ID: d627803f4438468c9cbb4bf822f8e903
I20240507 02:09:38.180034 12178 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240507 02:09:38.181490 12183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:38.171124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53082 (local address 127.31.131.126:34641)
0507 02:09:38.171465 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0507 02:09:38.171480 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0507 02:09:38.171510 (+ 30us) server_negotiation.cc:407] Connection header received
0507 02:09:38.171771 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:09:38.171777 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:09:38.171836 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:09:38.172937 (+ 1101us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:09:38.173178 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:38.174395 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:09:38.177742 (+ 3347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:38.177980 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:38.178075 (+ 95us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:09:38.178096 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:09:38.178097 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:09:38.178109 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:09:38.179967 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:09:38.180363 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:09:38.180366 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:09:38.180366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:09:38.180405 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:09:38.180532 (+ 127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:09:38.180533 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:09:38.180533 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:09:38.180730 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:09:38.181176 (+ 446us) server_negotiation.cc:1035] Waiting for connection context
0507 02:09:38.181241 (+ 65us) server_negotiation.cc:299] Negotiation successful
0507 02:09:38.181322 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":152,"threads_started":1}
I20240507 02:09:38.202793 12178 catalog_manager.cc:1370] Generated new certificate authority record
I20240507 02:09:38.203469 12178 catalog_manager.cc:1504] Loading token signing keys...
I20240507 02:09:38.223529 12178 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 902d0b3c51a44c65b0d2f7e6ffe9a301: Generated new TSK 0
I20240507 02:09:38.223738 12178 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.131.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab.
Entry for principal kudu/127.31.131.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab.
I20240507 02:09:38.833190 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.131.84:35369/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.65:0
--local_ip_for_outbound_sockets=127.31.131.65
--webserver_interface=127.31.131.65
--webserver_port=0
--tserver_master_addrs=127.31.131.126:34641
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab
--principal=kudu/127.31.131.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:37949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:09:38.935040 12206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:09:38.935199 12206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:09:38.935226 12206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:09:38.936488 12206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:09:38.936533 12206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:09:38.936566 12206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.65
I20240507 02:09:38.938310 12206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:37949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.31.131.84:35369/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab
--principal=kudu/127.31.131.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.131.65
--webserver_port=0
--tserver_master_addrs=127.31.131.126:34641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12206
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.65
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:09:38.938495 12206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:09:38.938701 12206 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:09:38.942346 12212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:09:38.942394 12214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:09:38.942477 12211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:09:38.942852 12206 server_base.cc:1034] running on GCE node
I20240507 02:09:38.942996 12206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:09:38.943171 12206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:09:38.944324 12206 hybrid_clock.cc:648] HybridClock initialized: now 1715047778944312 us; error 31 us; skew 500 ppm
May 07 02:09:38 dist-test-slave-v0bg krb5kdc[11649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047778, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:09:38.946694 12206 init.cc:377] Logged in from keytab as kudu/127.31.131.65@KRBTEST.COM (short username kudu)
I20240507 02:09:38.947099 12206 webserver.cc:460] Webserver started at http://127.31.131.65:46647/ using document root <none> and password file <none>
I20240507 02:09:38.947271 12206 fs_manager.cc:359] Metadata directory not provided
I20240507 02:09:38.947317 12206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:09:38.947412 12206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
May 07 02:09:38 dist-test-slave-v0bg krb5kdc[11649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047778, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.65@KRBTEST.COM for HTTP/127.31.131.84@KRBTEST.COM
I20240507 02:09:38.958461 12206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-0/data/instance:
uuid: "322a5d3259b84784a9ab75d487df2513"
format_stamp: "Formatted at 2024-05-07 02:09:38 on dist-test-slave-v0bg"
server_key: "501d46f59df567501d608671d7aaaa52"
server_key_iv: "7760d2f59c182ec1d27bf899b226d7d2"
server_key_version: "kuduclusterkey@0"
I20240507 02:09:38.958909 12206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-0/wal/instance:
uuid: "322a5d3259b84784a9ab75d487df2513"
format_stamp: "Formatted at 2024-05-07 02:09:38 on dist-test-slave-v0bg"
server_key: "501d46f59df567501d608671d7aaaa52"
server_key_iv: "7760d2f59c182ec1d27bf899b226d7d2"
server_key_version: "kuduclusterkey@0"
I20240507 02:09:38.960409 12206 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240507 02:09:38.973176 12221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:09:38.973393 12206 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240507 02:09:38.973512 12206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-0/wal
uuid: "322a5d3259b84784a9ab75d487df2513"
format_stamp: "Formatted at 2024-05-07 02:09:38 on dist-test-slave-v0bg"
server_key: "501d46f59df567501d608671d7aaaa52"
server_key_iv: "7760d2f59c182ec1d27bf899b226d7d2"
server_key_version: "kuduclusterkey@0"
I20240507 02:09:38.973618 12206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:09:38.995934 12206 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:09:38.998127 12206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:09:38.998298 12206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:09:38.998605 12206 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:09:38.998997 12206 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:09:38.999087 12206 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:09:38.999190 12206 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:09:38.999265 12206 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:09:39.005167 12206 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.65:35639
I20240507 02:09:39.005566 12206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
I20240507 02:09:39.008666 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 12206
I20240507 02:09:39.008774 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-0/wal/instance
I20240507 02:09:39.008953 32269 external_mini_cluster.cc:1428] Setting key 7a376cdfb7df4d7a374aac5bfd808078
I20240507 02:09:39.011179 12336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.65:35639 every 8 connection(s)
May 07 02:09:39 dist-test-slave-v0bg krb5kdc[11649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047778, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.65@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:09:39.020932 12339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:39.011814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:34641 (local address 127.31.131.65:57203)
0507 02:09:39.012205 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:39.012234 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0507 02:09:39.013202 (+ 968us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:09:39.013594 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:09:39.013599 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:09:39.013815 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:09:39.014192 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:39.014197 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:39.015255 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:09:39.015256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:09:39.015871 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:39.015874 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:39.016001 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:39.016434 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:09:39.016453 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:09:39.017892 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:09:39.019679 (+ 1787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:09:39.019684 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:09:39.019686 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:09:39.019923 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:09:39.020090 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:09:39.020091 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:09:39.020091 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:09:39.020161 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:09:39.020401 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:09:39.020406 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:09:39.020534 (+ 128us) client_negotiation.cc:769] Sending connection context
0507 02:09:39.020641 (+ 107us) client_negotiation.cc:240] Negotiation successful
0507 02:09:39.020758 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":85,"threads_started":1}
I20240507 02:09:39.021394 12340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:39.012073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:57203 (local address 127.31.131.126:34641)
0507 02:09:39.013398 (+ 1325us) server_negotiation.cc:206] Beginning negotiation
0507 02:09:39.013400 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:09:39.013411 (+ 11us) server_negotiation.cc:407] Connection header received
0507 02:09:39.013432 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:09:39.013432 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:09:39.013444 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:09:39.013513 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:09:39.014296 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:39.015157 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:09:39.015958 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:39.016138 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:39.017989 (+ 1851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:09:39.017994 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:09:39.017994 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:09:39.018001 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:09:39.019591 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:09:39.020001 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:09:39.020002 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:09:39.020002 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:09:39.020029 (+ 27us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:09:39.020224 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:09:39.020225 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:09:39.020226 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:09:39.020322 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:09:39.020378 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0507 02:09:39.021221 (+ 843us) server_negotiation.cc:299] Negotiation successful
0507 02:09:39.021304 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1298,"thread_start_us":77,"threads_started":1}
I20240507 02:09:39.021679 12337 heartbeater.cc:347] Connected to a master server at 127.31.131.126:34641
I20240507 02:09:39.021771 12337 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.31.131.66@KRBTEST.COM; defaulting to no policy
I20240507 02:09:39.021958 12337 heartbeater.cc:511] Master 127.31.131.126:34641 requested a full tablet report, sending...
I20240507 02:09:39.022485 12125 ts_manager.cc:194] Registered new tserver with Master: 322a5d3259b84784a9ab75d487df2513 (127.31.131.65:35639)
I20240507 02:09:39.023456 12125 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.131.65@KRBTEST.COM'} at 127.31.131.65:57203
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab.
Entry for principal kudu/127.31.131.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab.
I20240507 02:09:39.037370 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.131.84:35369/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.66:0
--local_ip_for_outbound_sockets=127.31.131.66
--webserver_interface=127.31.131.66
--webserver_port=0
--tserver_master_addrs=127.31.131.126:34641
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab
--principal=kudu/127.31.131.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:37949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:09:39.143716 12343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:09:39.143924 12343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:09:39.143958 12343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:09:39.145778 12343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:09:39.145833 12343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:09:39.145869 12343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.66
I20240507 02:09:39.147951 12343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:37949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.31.131.84:35369/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab
--principal=kudu/127.31.131.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.131.66
--webserver_port=0
--tserver_master_addrs=127.31.131.126:34641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12343
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.66
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:09:39.148144 12343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:09:39.148384 12343 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:09:39.152729 12353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:09:39.152725 12350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:09:39.153007 12351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:09:39.153493 12343 server_base.cc:1034] running on GCE node
I20240507 02:09:39.153648 12343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:09:39.153846 12343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:09:39.154985 12343 hybrid_clock.cc:648] HybridClock initialized: now 1715047779154966 us; error 35 us; skew 500 ppm
May 07 02:09:39 dist-test-slave-v0bg krb5kdc[11649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047779, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:09:39.157284 12343 init.cc:377] Logged in from keytab as kudu/127.31.131.66@KRBTEST.COM (short username kudu)
I20240507 02:09:39.157660 12343 webserver.cc:460] Webserver started at http://127.31.131.66:45905/ using document root <none> and password file <none>
I20240507 02:09:39.157850 12343 fs_manager.cc:359] Metadata directory not provided
I20240507 02:09:39.157893 12343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:09:39.157999 12343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
May 07 02:09:39 dist-test-slave-v0bg krb5kdc[11649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047779, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.66@KRBTEST.COM for HTTP/127.31.131.84@KRBTEST.COM
May 07 02:09:39 dist-test-slave-v0bg krb5kdc[11649](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1715047779, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:09:39.173053 12343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-1/data/instance:
uuid: "78ed6a7374fc41cd8f80970532ee24cb"
format_stamp: "Formatted at 2024-05-07 02:09:39 on dist-test-slave-v0bg"
server_key: "c4a5753ac0f3d5e6dea4530ece05815d"
server_key_iv: "4924193e44ad71efe1490076ad1113cc"
server_key_version: "kuduclusterkey@0"
I20240507 02:09:39.173501 12343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-1/wal/instance:
uuid: "78ed6a7374fc41cd8f80970532ee24cb"
format_stamp: "Formatted at 2024-05-07 02:09:39 on dist-test-slave-v0bg"
server_key: "c4a5753ac0f3d5e6dea4530ece05815d"
server_key_iv: "4924193e44ad71efe1490076ad1113cc"
server_key_version: "kuduclusterkey@0"
I20240507 02:09:39.176503 12343 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.000s
I20240507 02:09:39.203083 12360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:09:39.203306 12343 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.000s
I20240507 02:09:39.203418 12343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-1/wal
uuid: "78ed6a7374fc41cd8f80970532ee24cb"
format_stamp: "Formatted at 2024-05-07 02:09:39 on dist-test-slave-v0bg"
server_key: "c4a5753ac0f3d5e6dea4530ece05815d"
server_key_iv: "4924193e44ad71efe1490076ad1113cc"
server_key_version: "kuduclusterkey@0"
I20240507 02:09:39.203541 12343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:09:39.233969 12343 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:09:39.236213 12343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:09:39.236323 12343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:09:39.236495 12343 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:09:39.236850 12343 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:09:39.236891 12343 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:09:39.236917 12343 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:09:39.236932 12343 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:09:39.243058 12343 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.66:39789
I20240507 02:09:39.243439 12343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
I20240507 02:09:39.244370 12474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.66:39789 every 8 connection(s)
I20240507 02:09:39.251858 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 12343
I20240507 02:09:39.251968 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-1/wal/instance
I20240507 02:09:39.252130 32269 external_mini_cluster.cc:1428] Setting key ee8f5f10ead9ffccf48e7924e42fab77
May 07 02:09:39 dist-test-slave-v0bg krb5kdc[11649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047779, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.66@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:09:39.261933 12340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:39.245781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.66:51171 (local address 127.31.131.126:34641)
0507 02:09:39.245833 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0507 02:09:39.245835 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:09:39.246651 (+ 816us) server_negotiation.cc:407] Connection header received
0507 02:09:39.247833 (+ 1182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:09:39.247835 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:09:39.247852 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:09:39.247884 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:09:39.249034 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:39.249804 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:09:39.250603 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:39.250787 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:39.258460 (+ 7673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:09:39.258464 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:09:39.258465 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:09:39.258472 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:09:39.260256 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:09:39.260962 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:09:39.260964 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:09:39.260964 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:09:39.260992 (+ 28us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:09:39.261236 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:09:39.261237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:09:39.261237 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:09:39.261344 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:09:39.261389 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0507 02:09:39.261743 (+ 354us) server_negotiation.cc:299] Negotiation successful
0507 02:09:39.261833 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20240507 02:09:39.262585 12477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:39.244886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:34641 (local address 127.31.131.66:51171)
0507 02:09:39.246514 (+ 1628us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:39.246566 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0507 02:09:39.247695 (+ 1129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:09:39.248057 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:09:39.248061 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:09:39.248288 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:09:39.248920 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:39.248926 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:39.249928 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:09:39.249929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:09:39.250479 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:39.250481 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:39.250914 (+ 433us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:39.251410 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:09:39.251429 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:09:39.258340 (+ 6911us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:09:39.260358 (+ 2018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:09:39.260363 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:09:39.260366 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:09:39.260863 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:09:39.261065 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:09:39.261066 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:09:39.261066 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:09:39.261148 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:09:39.261431 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:09:39.261436 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:09:39.261570 (+ 134us) client_negotiation.cc:769] Sending connection context
0507 02:09:39.262266 (+ 696us) client_negotiation.cc:240] Negotiation successful
0507 02:09:39.262384 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1587,"thread_start_us":86,"threads_started":1}
I20240507 02:09:39.263208 12475 heartbeater.cc:347] Connected to a master server at 127.31.131.126:34641
I20240507 02:09:39.263446 12475 heartbeater.cc:464] Registering TS with master...
I20240507 02:09:39.263691 12475 heartbeater.cc:511] Master 127.31.131.126:34641 requested a full tablet report, sending...
I20240507 02:09:39.264065 12125 ts_manager.cc:194] Registered new tserver with Master: 78ed6a7374fc41cd8f80970532ee24cb (127.31.131.66:39789)
I20240507 02:09:39.264524 12125 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.131.66@KRBTEST.COM'} at 127.31.131.66:51171
WARNING: no policy specified for kudu/127.31.131.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab.
Entry for principal kudu/127.31.131.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab.
I20240507 02:09:39.291402 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.131.84:35369/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.67:0
--local_ip_for_outbound_sockets=127.31.131.67
--webserver_interface=127.31.131.67
--webserver_port=0
--tserver_master_addrs=127.31.131.126:34641
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab
--principal=kudu/127.31.131.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:37949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:09:39.401648 12480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:09:39.401949 12480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:09:39.402043 12480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:09:39.403985 12480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:09:39.404093 12480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:09:39.404181 12480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.67
I20240507 02:09:39.406299 12480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:37949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.31.131.84:35369/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab
--principal=kudu/127.31.131.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.131.67
--webserver_port=0
--tserver_master_addrs=127.31.131.126:34641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.67
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:09:39.406809 12480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:09:39.407073 12480 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:09:39.411782 12485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:09:39.411819 12486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:09:39.412539 12480 server_base.cc:1034] running on GCE node
W20240507 02:09:39.412859 12488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:09:39.413094 12480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:09:39.413301 12480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:09:39.414454 12480 hybrid_clock.cc:648] HybridClock initialized: now 1715047779414425 us; error 47 us; skew 500 ppm
May 07 02:09:39 dist-test-slave-v0bg krb5kdc[11649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047779, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:09:39.420843 12480 init.cc:377] Logged in from keytab as kudu/127.31.131.67@KRBTEST.COM (short username kudu)
I20240507 02:09:39.421205 12480 webserver.cc:460] Webserver started at http://127.31.131.67:44405/ using document root <none> and password file <none>
I20240507 02:09:39.421350 12480 fs_manager.cc:359] Metadata directory not provided
I20240507 02:09:39.421387 12480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:09:39.421483 12480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
May 07 02:09:39 dist-test-slave-v0bg krb5kdc[11649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047779, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.67@KRBTEST.COM for HTTP/127.31.131.84@KRBTEST.COM
I20240507 02:09:39.433864 12480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-2/data/instance:
uuid: "7c6b66e4a60041ecbd1811f7be53aade"
format_stamp: "Formatted at 2024-05-07 02:09:39 on dist-test-slave-v0bg"
server_key: "08da03e624ef15ffeebe63771eae5131"
server_key_iv: "169ac557df814bff4e034df985f1d7e7"
server_key_version: "kuduclusterkey@0"
I20240507 02:09:39.434218 12480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-2/wal/instance:
uuid: "7c6b66e4a60041ecbd1811f7be53aade"
format_stamp: "Formatted at 2024-05-07 02:09:39 on dist-test-slave-v0bg"
server_key: "08da03e624ef15ffeebe63771eae5131"
server_key_iv: "169ac557df814bff4e034df985f1d7e7"
server_key_version: "kuduclusterkey@0"
I20240507 02:09:39.436336 12480 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20240507 02:09:39.450655 12495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:09:39.450829 12480 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s
I20240507 02:09:39.450896 12480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-2/wal
uuid: "7c6b66e4a60041ecbd1811f7be53aade"
format_stamp: "Formatted at 2024-05-07 02:09:39 on dist-test-slave-v0bg"
server_key: "08da03e624ef15ffeebe63771eae5131"
server_key_iv: "169ac557df814bff4e034df985f1d7e7"
server_key_version: "kuduclusterkey@0"
I20240507 02:09:39.450951 12480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:09:39.469826 12480 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:09:39.472035 12480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:09:39.472191 12480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:09:39.472396 12480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:09:39.472741 12480 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:09:39.472820 12480 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:09:39.472887 12480 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:09:39.472939 12480 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:09:39.479054 12480 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.67:33619
I20240507 02:09:39.479135 12608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.67:33619 every 8 connection(s)
I20240507 02:09:39.479471 12480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
May 07 02:09:39 dist-test-slave-v0bg krb5kdc[11649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047779, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.67@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:09:39.487473 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 12480
I20240507 02:09:39.487571 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-2/wal/instance
I20240507 02:09:39.487735 32269 external_mini_cluster.cc:1428] Setting key 22f029cc0ec53fd5c494495d34847b1b
I20240507 02:09:39.490594 12340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:39.480161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.67:35987 (local address 127.31.131.126:34641)
0507 02:09:39.480216 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0507 02:09:39.480218 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:09:39.480381 (+ 163us) server_negotiation.cc:407] Connection header received
0507 02:09:39.481468 (+ 1087us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:09:39.481470 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:09:39.481482 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:09:39.481543 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:09:39.482344 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:39.483159 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:09:39.483926 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:39.484101 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:39.486240 (+ 2139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:09:39.486244 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:09:39.486244 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:09:39.486250 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:09:39.489167 (+ 2917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:09:39.489621 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:09:39.489623 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:09:39.489623 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:09:39.489649 (+ 26us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:09:39.489875 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:09:39.489876 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:09:39.489877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:09:39.489975 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:09:39.490023 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0507 02:09:39.490387 (+ 364us) server_negotiation.cc:299] Negotiation successful
0507 02:09:39.490474 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20240507 02:09:39.491003 12611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:39.480100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:34641 (local address 127.31.131.67:35987)
0507 02:09:39.480263 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:39.480295 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0507 02:09:39.481336 (+ 1041us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:09:39.481613 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:09:39.481617 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:09:39.481846 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:09:39.482238 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:39.482244 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:39.483266 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:09:39.483267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:09:39.483810 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:39.483813 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:39.483970 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:39.484607 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:09:39.484626 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:09:39.486122 (+ 1496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:09:39.489277 (+ 3155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:09:39.489283 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:09:39.489286 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:09:39.489531 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:09:39.489722 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:09:39.489722 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:09:39.489723 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:09:39.489799 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:09:39.490073 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:09:39.490078 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:09:39.490222 (+ 144us) client_negotiation.cc:769] Sending connection context
0507 02:09:39.490720 (+ 498us) client_negotiation.cc:240] Negotiation successful
0507 02:09:39.490833 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":76,"threads_started":1}
I20240507 02:09:39.491408 12609 heartbeater.cc:347] Connected to a master server at 127.31.131.126:34641
I20240507 02:09:39.491488 12609 heartbeater.cc:464] Registering TS with master...
I20240507 02:09:39.491680 12609 heartbeater.cc:511] Master 127.31.131.126:34641 requested a full tablet report, sending...
I20240507 02:09:39.492023 12125 ts_manager.cc:194] Registered new tserver with Master: 7c6b66e4a60041ecbd1811f7be53aade (127.31.131.67:33619)
I20240507 02:09:39.492628 12125 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.131.67@KRBTEST.COM'} at 127.31.131.67:35987
I20240507 02:09:39.499935 32269 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240507 02:09:39.500613 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 12206
I20240507 02:09:39.512028 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 12343
I20240507 02:09:39.517259 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 12480
I20240507 02:09:39.524338 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 12093
I20240507 02:09:39.531994 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:34641
--webserver_interface=127.31.131.126
--webserver_port=37149
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab
--principal=kudu/127.31.131.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:37949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.131.126:34641
--ranger_config_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.131.84:35369/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.31.131.84:35369/kms with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:09:39.608223 12612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:09:39.608376 12612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:09:39.608395 12612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:09:39.609612 12612 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240507 02:09:39.609651 12612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:09:39.609663 12612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:09:39.609688 12612 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240507 02:09:39.609711 12612 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240507 02:09:39.611189 12612 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:37949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.31.131.84:35369/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.31.131.126:34641
--ranger_config_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab
--principal=kudu/127.31.131.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:34641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.131.126
--webserver_port=37149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:09:39.611343 12612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:09:39.611496 12612 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240507 02:09:39.614861 12612 server_base.cc:1034] running on GCE node
W20240507 02:09:39.614849 12618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:09:39.614850 12620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:09:39.615075 12617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:09:39.615271 12612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:09:39.615470 12612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:09:39.616585 12612 hybrid_clock.cc:648] HybridClock initialized: now 1715047779616569 us; error 28 us; skew 500 ppm
May 07 02:09:39 dist-test-slave-v0bg krb5kdc[11649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047779, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:09:39.618634 12612 init.cc:377] Logged in from keytab as kudu/127.31.131.126@KRBTEST.COM (short username kudu)
I20240507 02:09:39.618968 12612 webserver.cc:460] Webserver started at http://127.31.131.126:37149/ using document root <none> and password file <none>
I20240507 02:09:39.619102 12612 fs_manager.cc:359] Metadata directory not provided
I20240507 02:09:39.619148 12612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
May 07 02:09:39 dist-test-slave-v0bg krb5kdc[11649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047779, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.126@KRBTEST.COM for HTTP/127.31.131.84@KRBTEST.COM
I20240507 02:09:39.645946 12612 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20240507 02:09:39.646648 12628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:09:39.646836 12612 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20240507 02:09:39.646944 12612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/master-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/master-0/wal
uuid: "902d0b3c51a44c65b0d2f7e6ffe9a301"
format_stamp: "Formatted at 2024-05-07 02:09:38 on dist-test-slave-v0bg"
server_key: "169afc9e7cc8e673864f33df8fc7462f"
server_key_iv: "2c8bed8796ed2270c2f1f4d2a751d07b"
server_key_version: "kuduclusterkey@0"
I20240507 02:09:39.647241 12612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:09:39.655769 12612 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:09:39.657356 12612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:09:39.657454 12612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:09:39.661157 12612 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.126:34641
I20240507 02:09:39.661206 12680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.126:34641 every 8 connection(s)
I20240507 02:09:39.661459 12612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
I20240507 02:09:39.662429 12681 sys_catalog.cc:263] Verifying existing consensus state
I20240507 02:09:39.662945 12681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 902d0b3c51a44c65b0d2f7e6ffe9a301: Bootstrap starting.
I20240507 02:09:39.666316 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 12612
I20240507 02:09:39.666628 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.131.84:35369/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.65:35639
--local_ip_for_outbound_sockets=127.31.131.65
--tserver_master_addrs=127.31.131.126:34641
--webserver_port=46647
--webserver_interface=127.31.131.65
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab
--principal=kudu/127.31.131.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:37949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.31.131.84:35369/kms with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240507 02:09:39.671339 12681 log.cc:824] T 00000000000000000000000000000000 P 902d0b3c51a44c65b0d2f7e6ffe9a301: Log is configured to *not* fsync() on all Append() calls
I20240507 02:09:39.672876 12681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 902d0b3c51a44c65b0d2f7e6ffe9a301: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240507 02:09:39.673110 12681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 902d0b3c51a44c65b0d2f7e6ffe9a301: Bootstrap complete.
I20240507 02:09:39.675065 12681 raft_consensus.cc:348] T 00000000000000000000000000000000 P 902d0b3c51a44c65b0d2f7e6ffe9a301 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "902d0b3c51a44c65b0d2f7e6ffe9a301" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 34641 } }
I20240507 02:09:39.675248 12681 raft_consensus.cc:729] T 00000000000000000000000000000000 P 902d0b3c51a44c65b0d2f7e6ffe9a301 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 902d0b3c51a44c65b0d2f7e6ffe9a301, State: Initialized, Role: FOLLOWER
I20240507 02:09:39.675361 12681 consensus_queue.cc:260] T 00000000000000000000000000000000 P 902d0b3c51a44c65b0d2f7e6ffe9a301 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "902d0b3c51a44c65b0d2f7e6ffe9a301" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 34641 } }
I20240507 02:09:39.675427 12681 raft_consensus.cc:388] T 00000000000000000000000000000000 P 902d0b3c51a44c65b0d2f7e6ffe9a301 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 02:09:39.675455 12681 raft_consensus.cc:482] T 00000000000000000000000000000000 P 902d0b3c51a44c65b0d2f7e6ffe9a301 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 02:09:39.675496 12681 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 902d0b3c51a44c65b0d2f7e6ffe9a301 [term 1 FOLLOWER]: Advancing to term 2
I20240507 02:09:39.676266 12681 raft_consensus.cc:504] T 00000000000000000000000000000000 P 902d0b3c51a44c65b0d2f7e6ffe9a301 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "902d0b3c51a44c65b0d2f7e6ffe9a301" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 34641 } }
I20240507 02:09:39.676354 12681 leader_election.cc:302] T 00000000000000000000000000000000 P 902d0b3c51a44c65b0d2f7e6ffe9a301 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 902d0b3c51a44c65b0d2f7e6ffe9a301; no voters:
I20240507 02:09:39.676482 12681 leader_election.cc:288] T 00000000000000000000000000000000 P 902d0b3c51a44c65b0d2f7e6ffe9a301 [CANDIDATE]: Term 2 election: Requested vote from peers
I20240507 02:09:39.676537 12685 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 902d0b3c51a44c65b0d2f7e6ffe9a301 [term 2 FOLLOWER]: Leader election won for term 2
I20240507 02:09:39.676669 12685 raft_consensus.cc:686] T 00000000000000000000000000000000 P 902d0b3c51a44c65b0d2f7e6ffe9a301 [term 2 LEADER]: Becoming Leader. State: Replica: 902d0b3c51a44c65b0d2f7e6ffe9a301, State: Running, Role: LEADER
I20240507 02:09:39.676682 12681 sys_catalog.cc:564] T 00000000000000000000000000000000 P 902d0b3c51a44c65b0d2f7e6ffe9a301 [sys.catalog]: configured and running, proceeding with master startup.
I20240507 02:09:39.676774 12685 consensus_queue.cc:237] T 00000000000000000000000000000000 P 902d0b3c51a44c65b0d2f7e6ffe9a301 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "902d0b3c51a44c65b0d2f7e6ffe9a301" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 34641 } }
I20240507 02:09:39.677008 12685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 902d0b3c51a44c65b0d2f7e6ffe9a301 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "902d0b3c51a44c65b0d2f7e6ffe9a301" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "902d0b3c51a44c65b0d2f7e6ffe9a301" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 34641 } } }
I20240507 02:09:39.677076 12685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 902d0b3c51a44c65b0d2f7e6ffe9a301 [sys.catalog]: This master's current role is: LEADER
I20240507 02:09:39.677186 12685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 902d0b3c51a44c65b0d2f7e6ffe9a301 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 902d0b3c51a44c65b0d2f7e6ffe9a301. Latest consensus state: current_term: 2 leader_uuid: "902d0b3c51a44c65b0d2f7e6ffe9a301" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "902d0b3c51a44c65b0d2f7e6ffe9a301" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 34641 } } }
I20240507 02:09:39.677232 12685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 902d0b3c51a44c65b0d2f7e6ffe9a301 [sys.catalog]: This master's current role is: LEADER
I20240507 02:09:39.677498 12690 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 02:09:39.677902 12690 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240507 02:09:39.678016 12690 catalog_manager.cc:1259] Loaded cluster ID: d627803f4438468c9cbb4bf822f8e903
I20240507 02:09:39.678047 12690 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240507 02:09:39.678175 12681 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20240507 02:09:39.679307 12690 catalog_manager.cc:1504] Loading token signing keys...
I20240507 02:09:39.679481 12690 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 902d0b3c51a44c65b0d2f7e6ffe9a301: Loaded TSK: 0
I20240507 02:09:39.679680 12690 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20240507 02:09:39.749212 12683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:09:39.749495 12683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:09:39.749531 12683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:09:39.751525 12683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:09:39.751580 12683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:09:39.751616 12683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.65
I20240507 02:09:39.753726 12683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:37949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.31.131.84:35369/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab
--principal=kudu/127.31.131.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.65:35639
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.131.65
--webserver_port=46647
--tserver_master_addrs=127.31.131.126:34641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12683
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.65
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:09:39.753921 12683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:09:39.754105 12683 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:09:39.758375 12721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:09:39.758383 12719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:09:39.759892 12718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:09:39.760568 12683 server_base.cc:1034] running on GCE node
I20240507 02:09:39.760756 12683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:09:39.760998 12683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:09:39.762187 12683 hybrid_clock.cc:648] HybridClock initialized: now 1715047779762159 us; error 38 us; skew 500 ppm
May 07 02:09:39 dist-test-slave-v0bg krb5kdc[11649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047779, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:09:39.764972 12683 init.cc:377] Logged in from keytab as kudu/127.31.131.65@KRBTEST.COM (short username kudu)
I20240507 02:09:39.765434 12683 webserver.cc:460] Webserver started at http://127.31.131.65:46647/ using document root <none> and password file <none>
I20240507 02:09:39.765655 12683 fs_manager.cc:359] Metadata directory not provided
I20240507 02:09:39.765751 12683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
May 07 02:09:39 dist-test-slave-v0bg krb5kdc[11649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047779, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.65@KRBTEST.COM for HTTP/127.31.131.84@KRBTEST.COM
I20240507 02:09:39.791883 12683 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20240507 02:09:39.792480 12729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:09:39.792614 12683 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240507 02:09:39.792676 12683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-0/wal
uuid: "322a5d3259b84784a9ab75d487df2513"
format_stamp: "Formatted at 2024-05-07 02:09:38 on dist-test-slave-v0bg"
server_key: "501d46f59df567501d608671d7aaaa52"
server_key_iv: "7760d2f59c182ec1d27bf899b226d7d2"
server_key_version: "kuduclusterkey@0"
I20240507 02:09:39.793011 12683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:09:39.812955 12683 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:09:39.815027 12683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:09:39.815142 12683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:09:39.815335 12683 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:09:39.815639 12683 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:09:39.815683 12683 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:09:39.815711 12683 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:09:39.815732 12683 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:09:39.821709 12683 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.65:35639
I20240507 02:09:39.822072 12683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
I20240507 02:09:39.822733 12842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.65:35639 every 8 connection(s)
I20240507 02:09:39.823617 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 12683
I20240507 02:09:39.823922 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.131.84:35369/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.66:39789
--local_ip_for_outbound_sockets=127.31.131.66
--tserver_master_addrs=127.31.131.126:34641
--webserver_port=45905
--webserver_interface=127.31.131.66
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab
--principal=kudu/127.31.131.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:37949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.31.131.84:35369/kms with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
May 07 02:09:39 dist-test-slave-v0bg krb5kdc[11649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047779, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.65@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:09:39.836553 12845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:39.823297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:34641 (local address 127.31.131.65:46019)
0507 02:09:39.826676 (+ 3379us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:39.826708 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0507 02:09:39.827476 (+ 768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:09:39.828314 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:09:39.828318 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:09:39.828540 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:09:39.828913 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:39.828918 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:39.830353 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:09:39.830354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:09:39.830898 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:39.830901 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:39.830996 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:39.831569 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:09:39.831585 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:09:39.833049 (+ 1464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:09:39.835200 (+ 2151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:09:39.835204 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:09:39.835207 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:09:39.835435 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:09:39.835635 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:09:39.835636 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:09:39.835637 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:09:39.835704 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:09:39.836060 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:09:39.836064 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:09:39.836183 (+ 119us) client_negotiation.cc:769] Sending connection context
0507 02:09:39.836277 (+ 94us) client_negotiation.cc:240] Negotiation successful
0507 02:09:39.836379 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3340,"thread_start_us":75,"threads_started":1}
I20240507 02:09:39.837426 12846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:39.823359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:46019 (local address 127.31.131.126:34641)
0507 02:09:39.827675 (+ 4316us) server_negotiation.cc:206] Beginning negotiation
0507 02:09:39.827685 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0507 02:09:39.827705 (+ 20us) server_negotiation.cc:407] Connection header received
0507 02:09:39.827822 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:09:39.827828 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:09:39.827873 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:09:39.827995 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:09:39.829032 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:39.830247 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:09:39.831228 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:39.831447 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:39.833152 (+ 1705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:09:39.833178 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:09:39.833179 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:09:39.833205 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:09:39.835098 (+ 1893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:09:39.835520 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:09:39.835522 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:09:39.835523 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:09:39.835569 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:09:39.835771 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:09:39.835772 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:09:39.835772 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:09:39.835987 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:09:39.837062 (+ 1075us) server_negotiation.cc:1035] Waiting for connection context
0507 02:09:39.837134 (+ 72us) server_negotiation.cc:299] Negotiation successful
0507 02:09:39.837243 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4274,"thread_start_us":75,"threads_started":1}
I20240507 02:09:39.838356 12843 heartbeater.cc:347] Connected to a master server at 127.31.131.126:34641
I20240507 02:09:39.838434 12843 heartbeater.cc:464] Registering TS with master...
I20240507 02:09:39.838649 12843 heartbeater.cc:511] Master 127.31.131.126:34641 requested a full tablet report, sending...
W20240507 02:09:39.839416 12843 heartbeater.cc:644] Failed to heartbeat to 127.31.131.126:34641 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20240507 02:09:39.839474 12843 heartbeater.cc:464] Registering TS with master...
I20240507 02:09:39.839553 12843 heartbeater.cc:511] Master 127.31.131.126:34641 requested a full tablet report, sending...
I20240507 02:09:39.839942 12843 heartbeater.cc:464] Registering TS with master...
I20240507 02:09:39.840044 12843 heartbeater.cc:511] Master 127.31.131.126:34641 requested a full tablet report, sending...
W20240507 02:09:39.840363 12843 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20240507 02:09:39.927331 12847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:09:39.927500 12847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:09:39.927521 12847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:09:39.928839 12847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:09:39.928886 12847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:09:39.928918 12847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.66
I20240507 02:09:39.930661 12847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:37949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.31.131.84:35369/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab
--principal=kudu/127.31.131.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.66:39789
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.131.66
--webserver_port=45905
--tserver_master_addrs=127.31.131.126:34641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.66
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:09:39.930873 12847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:09:39.931061 12847 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:09:39.934901 12853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:09:39.935010 12855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:09:39.934898 12852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:09:39.937594 12847 server_base.cc:1034] running on GCE node
I20240507 02:09:39.937795 12847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:09:39.938050 12847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:09:39.939384 12847 hybrid_clock.cc:648] HybridClock initialized: now 1715047779939366 us; error 33 us; skew 500 ppm
May 07 02:09:39 dist-test-slave-v0bg krb5kdc[11649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047779, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:09:39.942358 12847 init.cc:377] Logged in from keytab as kudu/127.31.131.66@KRBTEST.COM (short username kudu)
I20240507 02:09:39.942862 12847 webserver.cc:460] Webserver started at http://127.31.131.66:45905/ using document root <none> and password file <none>
I20240507 02:09:39.943058 12847 fs_manager.cc:359] Metadata directory not provided
I20240507 02:09:39.943118 12847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
May 07 02:09:39 dist-test-slave-v0bg krb5kdc[11649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047779, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.66@KRBTEST.COM for HTTP/127.31.131.84@KRBTEST.COM
I20240507 02:09:39.957276 12847 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20240507 02:09:39.958400 12863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:09:39.958604 12847 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20240507 02:09:39.958693 12847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-1/wal
uuid: "78ed6a7374fc41cd8f80970532ee24cb"
format_stamp: "Formatted at 2024-05-07 02:09:39 on dist-test-slave-v0bg"
server_key: "c4a5753ac0f3d5e6dea4530ece05815d"
server_key_iv: "4924193e44ad71efe1490076ad1113cc"
server_key_version: "kuduclusterkey@0"
I20240507 02:09:39.959015 12847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:09:39.977326 12847 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:09:39.979463 12847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:09:39.979588 12847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:09:39.979804 12847 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:09:39.980144 12847 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:09:39.980183 12847 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:09:39.980207 12847 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:09:39.980221 12847 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:09:39.987002 12847 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.66:39789
I20240507 02:09:39.987073 12976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.66:39789 every 8 connection(s)
I20240507 02:09:39.987521 12847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
I20240507 02:09:39.988893 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 12847
I20240507 02:09:39.989188 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.131.84:35369/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.67:33619
--local_ip_for_outbound_sockets=127.31.131.67
--tserver_master_addrs=127.31.131.126:34641
--webserver_port=44405
--webserver_interface=127.31.131.67
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab
--principal=kudu/127.31.131.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:37949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.31.131.84:35369/kms with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
May 07 02:09:39 dist-test-slave-v0bg krb5kdc[11649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047779, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.66@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:09:40.003041 12846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:39.988276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.66:45295 (local address 127.31.131.126:34641)
0507 02:09:39.988324 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0507 02:09:39.988326 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:09:39.988547 (+ 221us) server_negotiation.cc:407] Connection header received
0507 02:09:39.989693 (+ 1146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:09:39.989694 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:09:39.989706 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:09:39.989737 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:09:39.990917 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:39.991692 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:09:39.992474 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:39.992643 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:40.000337 (+ 7694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:09:40.000343 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:09:40.000343 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:09:40.000352 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:09:40.001752 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:09:40.002125 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:09:40.002126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:09:40.002126 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:09:40.002153 (+ 27us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:09:40.002331 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:09:40.002332 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:09:40.002333 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:09:40.002431 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:09:40.002492 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0507 02:09:40.002830 (+ 338us) server_negotiation.cc:299] Negotiation successful
0507 02:09:40.002933 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20240507 02:09:40.003466 12979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:39.988260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:34641 (local address 127.31.131.66:45295)
0507 02:09:39.988429 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:39.988460 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0507 02:09:39.989556 (+ 1096us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:09:39.989916 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:09:39.989921 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:09:39.990164 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:09:39.990607 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:39.990612 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:39.991802 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:09:39.991803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:09:39.992379 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:39.992382 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:39.992765 (+ 383us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:39.993388 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:09:39.993410 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:09:40.000222 (+ 6812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:09:40.001838 (+ 1616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:09:40.001844 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:09:40.001848 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:09:40.002055 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:09:40.002215 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:09:40.002216 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:09:40.002217 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:09:40.002279 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:09:40.002485 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:09:40.002490 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:09:40.002651 (+ 161us) client_negotiation.cc:769] Sending connection context
0507 02:09:40.003159 (+ 508us) client_negotiation.cc:240] Negotiation successful
0507 02:09:40.003293 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":90,"threads_started":1}
I20240507 02:09:40.003907 12977 heartbeater.cc:347] Connected to a master server at 127.31.131.126:34641
I20240507 02:09:40.004007 12977 heartbeater.cc:464] Registering TS with master...
I20240507 02:09:40.004221 12977 heartbeater.cc:511] Master 127.31.131.126:34641 requested a full tablet report, sending...
W20240507 02:09:40.005117 12977 heartbeater.cc:644] Failed to heartbeat to 127.31.131.126:34641 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20240507 02:09:40.005183 12977 heartbeater.cc:464] Registering TS with master...
I20240507 02:09:40.005278 12977 heartbeater.cc:511] Master 127.31.131.126:34641 requested a full tablet report, sending...
I20240507 02:09:40.005857 12977 heartbeater.cc:464] Registering TS with master...
I20240507 02:09:40.005961 12977 heartbeater.cc:511] Master 127.31.131.126:34641 requested a full tablet report, sending...
W20240507 02:09:40.006253 12977 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20240507 02:09:40.102777 12980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:09:40.103008 12980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:09:40.103044 12980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:09:40.104936 12980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:09:40.105000 12980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:09:40.105037 12980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.67
I20240507 02:09:40.107219 12980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:37949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.31.131.84:35369/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab
--principal=kudu/127.31.131.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.67:33619
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.131.67
--webserver_port=44405
--tserver_master_addrs=127.31.131.126:34641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.67
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:09:40.107429 12980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:09:40.107645 12980 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:09:40.112123 12985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:09:40.112175 12986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:09:40.112339 12988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:09:40.112648 12980 server_base.cc:1034] running on GCE node
I20240507 02:09:40.112802 12980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:09:40.113010 12980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:09:40.114156 12980 hybrid_clock.cc:648] HybridClock initialized: now 1715047780114130 us; error 43 us; skew 500 ppm
May 07 02:09:40 dist-test-slave-v0bg krb5kdc[11649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047780, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:09:40.117102 12980 init.cc:377] Logged in from keytab as kudu/127.31.131.67@KRBTEST.COM (short username kudu)
I20240507 02:09:40.117551 12980 webserver.cc:460] Webserver started at http://127.31.131.67:44405/ using document root <none> and password file <none>
I20240507 02:09:40.117744 12980 fs_manager.cc:359] Metadata directory not provided
I20240507 02:09:40.117805 12980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
May 07 02:09:40 dist-test-slave-v0bg krb5kdc[11649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047780, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.67@KRBTEST.COM for HTTP/127.31.131.84@KRBTEST.COM
I20240507 02:09:40.131829 12980 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20240507 02:09:40.135103 12996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:09:40.135274 12980 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.000s
I20240507 02:09:40.135335 12980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-2/wal
uuid: "7c6b66e4a60041ecbd1811f7be53aade"
format_stamp: "Formatted at 2024-05-07 02:09:39 on dist-test-slave-v0bg"
server_key: "08da03e624ef15ffeebe63771eae5131"
server_key_iv: "169ac557df814bff4e034df985f1d7e7"
server_key_version: "kuduclusterkey@0"
I20240507 02:09:40.135596 12980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:09:40.168782 12980 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:09:40.170943 12980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:09:40.171065 12980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:09:40.171257 12980 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:09:40.171581 12980 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:09:40.171623 12980 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:09:40.171653 12980 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:09:40.171685 12980 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:09:40.178495 12980 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.67:33619
I20240507 02:09:40.178887 12980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
I20240507 02:09:40.180213 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 12980
I20240507 02:09:40.179257 13110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.67:33619 every 8 connection(s)
May 07 02:09:40 dist-test-slave-v0bg krb5kdc[11649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047780, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.67@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:09:40.191296 13115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:40.182135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53088 (local address 127.31.131.126:34641)
0507 02:09:40.183101 (+ 966us) server_negotiation.cc:206] Beginning negotiation
0507 02:09:40.183102 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:09:40.183112 (+ 10us) server_negotiation.cc:407] Connection header received
0507 02:09:40.183130 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:09:40.183131 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:09:40.183141 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:09:40.184315 (+ 1174us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:09:40.184349 (+ 34us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:40.185102 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:09:40.187130 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:40.187301 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:40.187340 (+ 39us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:09:40.187343 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:09:40.187343 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:09:40.187347 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:09:40.189964 (+ 2617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:09:40.190275 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:09:40.190276 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:09:40.190276 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:09:40.190298 (+ 22us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:09:40.190442 (+ 144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:09:40.190443 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:09:40.190443 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:09:40.190538 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:09:40.191074 (+ 536us) server_negotiation.cc:1035] Waiting for connection context
0507 02:09:40.191112 (+ 38us) server_negotiation.cc:299] Negotiation successful
0507 02:09:40.191188 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":941,"thread_start_us":58,"threads_started":1}
I20240507 02:09:40.195307 12846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:40.180222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.67:60697 (local address 127.31.131.126:34641)
0507 02:09:40.180275 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0507 02:09:40.180277 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:09:40.180288 (+ 11us) server_negotiation.cc:407] Connection header received
0507 02:09:40.181227 (+ 939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:09:40.181228 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:09:40.181243 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:09:40.181305 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:09:40.182140 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:40.182929 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:09:40.183700 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:40.183873 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:40.188239 (+ 4366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:09:40.188243 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:09:40.188243 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:09:40.188250 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:09:40.189813 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:09:40.190258 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:09:40.190260 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:09:40.190260 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:09:40.190286 (+ 26us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:09:40.194651 (+ 4365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:09:40.194653 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:09:40.194653 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:09:40.194759 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:09:40.194799 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0507 02:09:40.195133 (+ 334us) server_negotiation.cc:299] Negotiation successful
0507 02:09:40.195217 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20240507 02:09:40.195735 13113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:40.179960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:34641 (local address 127.31.131.67:60697)
0507 02:09:40.180110 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:40.180142 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0507 02:09:40.181066 (+ 924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:09:40.181423 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:09:40.181427 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:09:40.181641 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:09:40.182037 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:40.182042 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:40.183024 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:09:40.183078 (+ 54us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:09:40.183618 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:40.183621 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:40.183769 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:40.185506 (+ 1737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:09:40.185527 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:09:40.188131 (+ 2604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:09:40.189895 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:09:40.189900 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:09:40.189903 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:09:40.190161 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:09:40.190354 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:09:40.190355 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:09:40.190355 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:09:40.190435 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:09:40.194844 (+ 4409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:09:40.194850 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:09:40.194989 (+ 139us) client_negotiation.cc:769] Sending connection context
0507 02:09:40.195424 (+ 435us) client_negotiation.cc:240] Negotiation successful
0507 02:09:40.195540 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":88,"threads_started":1}
I20240507 02:09:40.196098 13111 heartbeater.cc:347] Connected to a master server at 127.31.131.126:34641
I20240507 02:09:40.196188 13111 heartbeater.cc:464] Registering TS with master...
I20240507 02:09:40.196388 13111 heartbeater.cc:511] Master 127.31.131.126:34641 requested a full tablet report, sending...
W20240507 02:09:40.196934 13111 heartbeater.cc:644] Failed to heartbeat to 127.31.131.126:34641 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20240507 02:09:40.197003 13111 heartbeater.cc:464] Registering TS with master...
I20240507 02:09:40.197089 13111 heartbeater.cc:511] Master 127.31.131.126:34641 requested a full tablet report, sending...
I20240507 02:09:40.197364 13111 heartbeater.cc:464] Registering TS with master...
I20240507 02:09:40.197468 13111 heartbeater.cc:511] Master 127.31.131.126:34641 requested a full tablet report, sending...
W20240507 02:09:40.197748 13111 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
May 07 02:09:40 dist-test-slave-v0bg krb5kdc[11649](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1715047780, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:09:40.840515 12843 heartbeater.cc:464] Registering TS with master...
I20240507 02:09:40.840683 12843 heartbeater.cc:511] Master 127.31.131.126:34641 requested a full tablet report, sending...
I20240507 02:09:40.840974 12645 ts_manager.cc:194] Registered new tserver with Master: 322a5d3259b84784a9ab75d487df2513 (127.31.131.65:35639)
I20240507 02:09:40.841675 12645 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.131.65@KRBTEST.COM'} at 127.31.131.65:46019
I20240507 02:09:41.006412 12977 heartbeater.cc:464] Registering TS with master...
I20240507 02:09:41.006603 12977 heartbeater.cc:511] Master 127.31.131.126:34641 requested a full tablet report, sending...
I20240507 02:09:41.006863 12645 ts_manager.cc:194] Registered new tserver with Master: 78ed6a7374fc41cd8f80970532ee24cb (127.31.131.66:39789)
I20240507 02:09:41.007241 12645 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.131.66@KRBTEST.COM'} at 127.31.131.66:45295
May 07 02:09:41 dist-test-slave-v0bg krb5kdc[11649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047780, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.126@KRBTEST.COM for HTTP/127.31.131.84@KRBTEST.COM
I20240507 02:09:41.197925 13111 heartbeater.cc:464] Registering TS with master...
I20240507 02:09:41.198108 13111 heartbeater.cc:511] Master 127.31.131.126:34641 requested a full tablet report, sending...
I20240507 02:09:41.198400 12645 ts_manager.cc:194] Registered new tserver with Master: 7c6b66e4a60041ecbd1811f7be53aade (127.31.131.67:33619)
I20240507 02:09:41.198976 12645 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.131.67@KRBTEST.COM'} at 127.31.131.67:60697
I20240507 02:09:41.201592 32269 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240507 02:09:41.211972 13132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:41.202807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53092 (local address 127.31.131.126:34641)
0507 02:09:41.205111 (+ 2304us) server_negotiation.cc:206] Beginning negotiation
0507 02:09:41.205114 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0507 02:09:41.205128 (+ 14us) server_negotiation.cc:407] Connection header received
0507 02:09:41.205149 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:09:41.205150 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:09:41.205162 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:09:41.205222 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:09:41.206070 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:41.206902 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:09:41.207819 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:41.207984 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:41.208892 (+ 908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:09:41.208895 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:09:41.208896 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:09:41.208902 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:09:41.210687 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:09:41.211045 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:09:41.211047 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:09:41.211047 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:09:41.211071 (+ 24us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:09:41.211200 (+ 129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:09:41.211200 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:09:41.211201 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:09:41.211295 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:09:41.211377 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0507 02:09:41.211770 (+ 393us) server_negotiation.cc:299] Negotiation successful
0507 02:09:41.211865 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2283,"thread_start_us":63,"threads_started":1}
I20240507 02:09:41.214286 12645 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53092:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 02:09:41.216356 12645 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 02:09:41.229117 13138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:41.222794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.65:35639 (local address 127.0.0.1:41506)
0507 02:09:41.225569 (+ 2775us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:41.225596 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0507 02:09:41.225656 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:09:41.225963 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:09:41.225966 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:09:41.225969 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:09:41.226228 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:41.226229 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:41.227930 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:09:41.227931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:09:41.228878 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:41.228880 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:41.228926 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:41.228958 (+ 32us) client_negotiation.cc:769] Sending connection context
0507 02:09:41.228997 (+ 39us) client_negotiation.cc:240] Negotiation successful
0507 02:09:41.229036 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2746,"thread_start_us":90,"threads_started":1}
I20240507 02:09:41.230646 13142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:41.223064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41506 (local address 127.31.131.65:35639)
0507 02:09:41.225777 (+ 2713us) server_negotiation.cc:206] Beginning negotiation
0507 02:09:41.225781 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0507 02:09:41.225791 (+ 10us) server_negotiation.cc:407] Connection header received
0507 02:09:41.225828 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:09:41.225829 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:09:41.225847 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:09:41.226302 (+ 455us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:09:41.226372 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:41.227844 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:09:41.229645 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:41.230148 (+ 503us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:41.230199 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0507 02:09:41.230233 (+ 34us) server_negotiation.cc:299] Negotiation successful
0507 02:09:41.230526 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2682,"thread_start_us":53,"threads_started":1}
I20240507 02:09:41.231299 12777 tablet_service.cc:1467] Processing CreateTablet for tablet dbb194f09b9f49298ea4cdb86f13dbb8 (DEFAULT_TABLE table=test-table [id=a5d56ef02bba41bd938ef261be808962]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:09:41.231606 12777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbb194f09b9f49298ea4cdb86f13dbb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:09:41.233321 13138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:41.222829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.66:39789 (local address 127.0.0.1:42150)
0507 02:09:41.229500 (+ 6671us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:41.229508 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0507 02:09:41.229540 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:09:41.230817 (+ 1277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:09:41.230818 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:09:41.230820 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:09:41.230976 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:41.230978 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:41.232302 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:09:41.232303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:09:41.233149 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:41.233151 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:41.233206 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:41.233209 (+ 3us) client_negotiation.cc:769] Sending connection context
0507 02:09:41.233230 (+ 21us) client_negotiation.cc:240] Negotiation successful
0507 02:09:41.233258 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6660,"thread_start_us":73,"threads_started":1}
I20240507 02:09:41.235378 13144 tablet_bootstrap.cc:492] T dbb194f09b9f49298ea4cdb86f13dbb8 P 322a5d3259b84784a9ab75d487df2513: Bootstrap starting.
I20240507 02:09:41.235985 13144 tablet_bootstrap.cc:654] T dbb194f09b9f49298ea4cdb86f13dbb8 P 322a5d3259b84784a9ab75d487df2513: Neither blocks nor log segments found. Creating new log.
I20240507 02:09:41.236306 13144 log.cc:824] T dbb194f09b9f49298ea4cdb86f13dbb8 P 322a5d3259b84784a9ab75d487df2513: Log is configured to *not* fsync() on all Append() calls
I20240507 02:09:41.237002 13144 tablet_bootstrap.cc:492] T dbb194f09b9f49298ea4cdb86f13dbb8 P 322a5d3259b84784a9ab75d487df2513: No bootstrap required, opened a new log
I20240507 02:09:41.237015 13143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:41.223104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45036 (local address 127.31.131.67:33619)
0507 02:09:41.223202 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0507 02:09:41.223206 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0507 02:09:41.233511 (+ 10305us) server_negotiation.cc:407] Connection header received
0507 02:09:41.233636 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:09:41.233638 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:09:41.233653 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:09:41.233703 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:09:41.234032 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:41.235140 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:09:41.236245 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:41.236718 (+ 473us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:41.236766 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0507 02:09:41.236890 (+ 124us) server_negotiation.cc:299] Negotiation successful
0507 02:09:41.236925 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71,"thread_start_us":50,"threads_started":1}
I20240507 02:09:41.237058 13144 ts_tablet_manager.cc:1367] T dbb194f09b9f49298ea4cdb86f13dbb8 P 322a5d3259b84784a9ab75d487df2513: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240507 02:09:41.237205 13138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:41.223027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.67:33619 (local address 127.0.0.1:45036)
0507 02:09:41.233454 (+ 10427us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:41.233463 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0507 02:09:41.233539 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:09:41.233796 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:09:41.233797 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:09:41.233798 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:09:41.233952 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:41.233953 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:41.235249 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:09:41.235250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:09:41.236119 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:41.236121 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:41.236795 (+ 674us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:41.236799 (+ 4us) client_negotiation.cc:769] Sending connection context
0507 02:09:41.237100 (+ 301us) client_negotiation.cc:240] Negotiation successful
0507 02:09:41.237136 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10417,"thread_start_us":48,"threads_started":1}
I20240507 02:09:41.237751 13027 tablet_service.cc:1467] Processing CreateTablet for tablet dbb194f09b9f49298ea4cdb86f13dbb8 (DEFAULT_TABLE table=test-table [id=a5d56ef02bba41bd938ef261be808962]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:09:41.238006 13027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbb194f09b9f49298ea4cdb86f13dbb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:09:41.238081 13140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:41.222913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42150 (local address 127.31.131.66:39789)
0507 02:09:41.230631 (+ 7718us) server_negotiation.cc:206] Beginning negotiation
0507 02:09:41.230635 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0507 02:09:41.230646 (+ 11us) server_negotiation.cc:407] Connection header received
0507 02:09:41.230683 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:09:41.230685 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:09:41.230702 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:09:41.231067 (+ 365us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:09:41.231140 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:41.232204 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:09:41.237404 (+ 5200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:41.237896 (+ 492us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:41.237933 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0507 02:09:41.237964 (+ 31us) server_negotiation.cc:299] Negotiation successful
0507 02:09:41.237993 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7685,"thread_start_us":41,"threads_started":1}
I20240507 02:09:41.238524 12900 tablet_service.cc:1467] Processing CreateTablet for tablet dbb194f09b9f49298ea4cdb86f13dbb8 (DEFAULT_TABLE table=test-table [id=a5d56ef02bba41bd938ef261be808962]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:09:41.238785 12900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbb194f09b9f49298ea4cdb86f13dbb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:09:41.239162 13144 raft_consensus.cc:348] T dbb194f09b9f49298ea4cdb86f13dbb8 P 322a5d3259b84784a9ab75d487df2513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "322a5d3259b84784a9ab75d487df2513" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35639 } } peers { permanent_uuid: "78ed6a7374fc41cd8f80970532ee24cb" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 39789 } } peers { permanent_uuid: "7c6b66e4a60041ecbd1811f7be53aade" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 33619 } }
I20240507 02:09:41.239289 13144 raft_consensus.cc:374] T dbb194f09b9f49298ea4cdb86f13dbb8 P 322a5d3259b84784a9ab75d487df2513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:09:41.239310 13144 raft_consensus.cc:729] T dbb194f09b9f49298ea4cdb86f13dbb8 P 322a5d3259b84784a9ab75d487df2513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 322a5d3259b84784a9ab75d487df2513, State: Initialized, Role: FOLLOWER
I20240507 02:09:41.239410 13144 consensus_queue.cc:260] T dbb194f09b9f49298ea4cdb86f13dbb8 P 322a5d3259b84784a9ab75d487df2513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "322a5d3259b84784a9ab75d487df2513" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35639 } } peers { permanent_uuid: "78ed6a7374fc41cd8f80970532ee24cb" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 39789 } } peers { permanent_uuid: "7c6b66e4a60041ecbd1811f7be53aade" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 33619 } }
I20240507 02:09:41.239645 13144 ts_tablet_manager.cc:1398] T dbb194f09b9f49298ea4cdb86f13dbb8 P 322a5d3259b84784a9ab75d487df2513: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240507 02:09:41.240017 12843 heartbeater.cc:503] Master 127.31.131.126:34641 was elected leader, sending a full tablet report...
I20240507 02:09:41.246450 13147 tablet_bootstrap.cc:492] T dbb194f09b9f49298ea4cdb86f13dbb8 P 7c6b66e4a60041ecbd1811f7be53aade: Bootstrap starting.
I20240507 02:09:41.247118 13147 tablet_bootstrap.cc:654] T dbb194f09b9f49298ea4cdb86f13dbb8 P 7c6b66e4a60041ecbd1811f7be53aade: Neither blocks nor log segments found. Creating new log.
I20240507 02:09:41.247462 13147 log.cc:824] T dbb194f09b9f49298ea4cdb86f13dbb8 P 7c6b66e4a60041ecbd1811f7be53aade: Log is configured to *not* fsync() on all Append() calls
I20240507 02:09:41.247825 13148 tablet_bootstrap.cc:492] T dbb194f09b9f49298ea4cdb86f13dbb8 P 78ed6a7374fc41cd8f80970532ee24cb: Bootstrap starting.
I20240507 02:09:41.248386 13148 tablet_bootstrap.cc:654] T dbb194f09b9f49298ea4cdb86f13dbb8 P 78ed6a7374fc41cd8f80970532ee24cb: Neither blocks nor log segments found. Creating new log.
I20240507 02:09:41.248683 13148 log.cc:824] T dbb194f09b9f49298ea4cdb86f13dbb8 P 78ed6a7374fc41cd8f80970532ee24cb: Log is configured to *not* fsync() on all Append() calls
I20240507 02:09:41.249476 13147 tablet_bootstrap.cc:492] T dbb194f09b9f49298ea4cdb86f13dbb8 P 7c6b66e4a60041ecbd1811f7be53aade: No bootstrap required, opened a new log
I20240507 02:09:41.249573 13147 ts_tablet_manager.cc:1367] T dbb194f09b9f49298ea4cdb86f13dbb8 P 7c6b66e4a60041ecbd1811f7be53aade: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240507 02:09:41.251588 13147 raft_consensus.cc:348] T dbb194f09b9f49298ea4cdb86f13dbb8 P 7c6b66e4a60041ecbd1811f7be53aade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "322a5d3259b84784a9ab75d487df2513" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35639 } } peers { permanent_uuid: "78ed6a7374fc41cd8f80970532ee24cb" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 39789 } } peers { permanent_uuid: "7c6b66e4a60041ecbd1811f7be53aade" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 33619 } }
I20240507 02:09:41.251777 13147 raft_consensus.cc:374] T dbb194f09b9f49298ea4cdb86f13dbb8 P 7c6b66e4a60041ecbd1811f7be53aade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:09:41.251837 13147 raft_consensus.cc:729] T dbb194f09b9f49298ea4cdb86f13dbb8 P 7c6b66e4a60041ecbd1811f7be53aade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c6b66e4a60041ecbd1811f7be53aade, State: Initialized, Role: FOLLOWER
I20240507 02:09:41.252260 13147 consensus_queue.cc:260] T dbb194f09b9f49298ea4cdb86f13dbb8 P 7c6b66e4a60041ecbd1811f7be53aade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "322a5d3259b84784a9ab75d487df2513" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35639 } } peers { permanent_uuid: "78ed6a7374fc41cd8f80970532ee24cb" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 39789 } } peers { permanent_uuid: "7c6b66e4a60041ecbd1811f7be53aade" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 33619 } }
I20240507 02:09:41.252570 13147 ts_tablet_manager.cc:1398] T dbb194f09b9f49298ea4cdb86f13dbb8 P 7c6b66e4a60041ecbd1811f7be53aade: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240507 02:09:41.252704 13111 heartbeater.cc:503] Master 127.31.131.126:34641 was elected leader, sending a full tablet report...
I20240507 02:09:41.255157 13148 tablet_bootstrap.cc:492] T dbb194f09b9f49298ea4cdb86f13dbb8 P 78ed6a7374fc41cd8f80970532ee24cb: No bootstrap required, opened a new log
I20240507 02:09:41.255266 13148 ts_tablet_manager.cc:1367] T dbb194f09b9f49298ea4cdb86f13dbb8 P 78ed6a7374fc41cd8f80970532ee24cb: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240507 02:09:41.257262 13148 raft_consensus.cc:348] T dbb194f09b9f49298ea4cdb86f13dbb8 P 78ed6a7374fc41cd8f80970532ee24cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "322a5d3259b84784a9ab75d487df2513" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35639 } } peers { permanent_uuid: "78ed6a7374fc41cd8f80970532ee24cb" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 39789 } } peers { permanent_uuid: "7c6b66e4a60041ecbd1811f7be53aade" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 33619 } }
I20240507 02:09:41.257457 13148 raft_consensus.cc:374] T dbb194f09b9f49298ea4cdb86f13dbb8 P 78ed6a7374fc41cd8f80970532ee24cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:09:41.257515 13148 raft_consensus.cc:729] T dbb194f09b9f49298ea4cdb86f13dbb8 P 78ed6a7374fc41cd8f80970532ee24cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78ed6a7374fc41cd8f80970532ee24cb, State: Initialized, Role: FOLLOWER
I20240507 02:09:41.257660 13148 consensus_queue.cc:260] T dbb194f09b9f49298ea4cdb86f13dbb8 P 78ed6a7374fc41cd8f80970532ee24cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "322a5d3259b84784a9ab75d487df2513" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35639 } } peers { permanent_uuid: "78ed6a7374fc41cd8f80970532ee24cb" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 39789 } } peers { permanent_uuid: "7c6b66e4a60041ecbd1811f7be53aade" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 33619 } }
I20240507 02:09:41.257943 13148 ts_tablet_manager.cc:1398] T dbb194f09b9f49298ea4cdb86f13dbb8 P 78ed6a7374fc41cd8f80970532ee24cb: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240507 02:09:41.258075 12977 heartbeater.cc:503] Master 127.31.131.126:34641 was elected leader, sending a full tablet report...
I20240507 02:09:41.285053 13152 raft_consensus.cc:482] T dbb194f09b9f49298ea4cdb86f13dbb8 P 78ed6a7374fc41cd8f80970532ee24cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 02:09:41.285261 13152 raft_consensus.cc:504] T dbb194f09b9f49298ea4cdb86f13dbb8 P 78ed6a7374fc41cd8f80970532ee24cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "322a5d3259b84784a9ab75d487df2513" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35639 } } peers { permanent_uuid: "78ed6a7374fc41cd8f80970532ee24cb" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 39789 } } peers { permanent_uuid: "7c6b66e4a60041ecbd1811f7be53aade" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 33619 } }
I20240507 02:09:41.286453 13152 leader_election.cc:288] T dbb194f09b9f49298ea4cdb86f13dbb8 P 78ed6a7374fc41cd8f80970532ee24cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 322a5d3259b84784a9ab75d487df2513 (127.31.131.65:35639), 7c6b66e4a60041ecbd1811f7be53aade (127.31.131.67:33619)
I20240507 02:09:41.295034 13142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:41.286037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.66:43801 (local address 127.31.131.65:35639)
0507 02:09:41.286080 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0507 02:09:41.286083 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0507 02:09:41.290198 (+ 4115us) server_negotiation.cc:407] Connection header received
0507 02:09:41.290299 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:09:41.290300 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:09:41.290310 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:09:41.290335 (+ 25us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:09:41.291331 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:41.292056 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:09:41.293911 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:41.294373 (+ 462us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:41.294383 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0507 02:09:41.294913 (+ 530us) server_negotiation.cc:299] Negotiation successful
0507 02:09:41.294963 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20240507 02:09:41.295471 13153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:41.285831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.65:35639 (local address 127.31.131.66:43801)
0507 02:09:41.290116 (+ 4285us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:41.290129 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0507 02:09:41.290230 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:09:41.290406 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:09:41.290407 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:09:41.290408 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:09:41.290711 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:41.290713 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:41.292147 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:09:41.292148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:09:41.293018 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:41.293021 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:41.294805 (+ 1784us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:41.294811 (+ 6us) client_negotiation.cc:769] Sending connection context
0507 02:09:41.295342 (+ 531us) client_negotiation.cc:240] Negotiation successful
0507 02:09:41.295392 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4257,"thread_start_us":101,"threads_started":1}
I20240507 02:09:41.295562 13143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:41.286396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.66:47399 (local address 127.31.131.67:33619)
0507 02:09:41.286433 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0507 02:09:41.286435 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:09:41.290857 (+ 4422us) server_negotiation.cc:407] Connection header received
0507 02:09:41.290942 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:09:41.290943 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:09:41.290952 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:09:41.290976 (+ 24us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:09:41.293120 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:41.293791 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:09:41.294880 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:41.295334 (+ 454us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:41.295342 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0507 02:09:41.295458 (+ 116us) server_negotiation.cc:299] Negotiation successful
0507 02:09:41.295492 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20240507 02:09:41.295928 13154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:41.286219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.67:33619 (local address 127.31.131.66:47399)
0507 02:09:41.290802 (+ 4583us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:41.290810 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0507 02:09:41.290879 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:09:41.291033 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:09:41.291034 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:09:41.291035 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:09:41.291251 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:41.291253 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:41.293902 (+ 2649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:09:41.293903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:09:41.294782 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:41.294784 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:41.295373 (+ 589us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:41.295376 (+ 3us) client_negotiation.cc:769] Sending connection context
0507 02:09:41.295819 (+ 443us) client_negotiation.cc:240] Negotiation successful
0507 02:09:41.295857 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4567,"thread_start_us":76,"threads_started":1}
I20240507 02:09:41.295926 12797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbb194f09b9f49298ea4cdb86f13dbb8" candidate_uuid: "78ed6a7374fc41cd8f80970532ee24cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "322a5d3259b84784a9ab75d487df2513" is_pre_election: true
I20240507 02:09:41.296070 12797 raft_consensus.cc:2454] T dbb194f09b9f49298ea4cdb86f13dbb8 P 322a5d3259b84784a9ab75d487df2513 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78ed6a7374fc41cd8f80970532ee24cb in term 0.
I20240507 02:09:41.296295 12866 leader_election.cc:302] T dbb194f09b9f49298ea4cdb86f13dbb8 P 78ed6a7374fc41cd8f80970532ee24cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 322a5d3259b84784a9ab75d487df2513, 78ed6a7374fc41cd8f80970532ee24cb; no voters:
I20240507 02:09:41.296383 13065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbb194f09b9f49298ea4cdb86f13dbb8" candidate_uuid: "78ed6a7374fc41cd8f80970532ee24cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c6b66e4a60041ecbd1811f7be53aade" is_pre_election: true
I20240507 02:09:41.296504 13065 raft_consensus.cc:2454] T dbb194f09b9f49298ea4cdb86f13dbb8 P 7c6b66e4a60041ecbd1811f7be53aade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78ed6a7374fc41cd8f80970532ee24cb in term 0.
I20240507 02:09:41.296826 13152 raft_consensus.cc:2781] T dbb194f09b9f49298ea4cdb86f13dbb8 P 78ed6a7374fc41cd8f80970532ee24cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 02:09:41.297051 13152 raft_consensus.cc:482] T dbb194f09b9f49298ea4cdb86f13dbb8 P 78ed6a7374fc41cd8f80970532ee24cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 02:09:41.297113 13152 raft_consensus.cc:3037] T dbb194f09b9f49298ea4cdb86f13dbb8 P 78ed6a7374fc41cd8f80970532ee24cb [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:09:41.297966 13152 raft_consensus.cc:504] T dbb194f09b9f49298ea4cdb86f13dbb8 P 78ed6a7374fc41cd8f80970532ee24cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "322a5d3259b84784a9ab75d487df2513" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35639 } } peers { permanent_uuid: "78ed6a7374fc41cd8f80970532ee24cb" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 39789 } } peers { permanent_uuid: "7c6b66e4a60041ecbd1811f7be53aade" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 33619 } }
I20240507 02:09:41.298251 12797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbb194f09b9f49298ea4cdb86f13dbb8" candidate_uuid: "78ed6a7374fc41cd8f80970532ee24cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "322a5d3259b84784a9ab75d487df2513"
I20240507 02:09:41.298311 12797 raft_consensus.cc:3037] T dbb194f09b9f49298ea4cdb86f13dbb8 P 322a5d3259b84784a9ab75d487df2513 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:09:41.299243 12797 raft_consensus.cc:2454] T dbb194f09b9f49298ea4cdb86f13dbb8 P 322a5d3259b84784a9ab75d487df2513 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78ed6a7374fc41cd8f80970532ee24cb in term 1.
I20240507 02:09:41.299456 12866 leader_election.cc:302] T dbb194f09b9f49298ea4cdb86f13dbb8 P 78ed6a7374fc41cd8f80970532ee24cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 322a5d3259b84784a9ab75d487df2513, 78ed6a7374fc41cd8f80970532ee24cb; no voters:
I20240507 02:09:41.299806 13065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbb194f09b9f49298ea4cdb86f13dbb8" candidate_uuid: "78ed6a7374fc41cd8f80970532ee24cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c6b66e4a60041ecbd1811f7be53aade"
I20240507 02:09:41.299929 13065 raft_consensus.cc:3037] T dbb194f09b9f49298ea4cdb86f13dbb8 P 7c6b66e4a60041ecbd1811f7be53aade [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:09:41.300748 13065 raft_consensus.cc:2454] T dbb194f09b9f49298ea4cdb86f13dbb8 P 7c6b66e4a60041ecbd1811f7be53aade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78ed6a7374fc41cd8f80970532ee24cb in term 1.
I20240507 02:09:41.300971 13152 leader_election.cc:288] T dbb194f09b9f49298ea4cdb86f13dbb8 P 78ed6a7374fc41cd8f80970532ee24cb [CANDIDATE]: Term 1 election: Requested vote from peers 322a5d3259b84784a9ab75d487df2513 (127.31.131.65:35639), 7c6b66e4a60041ecbd1811f7be53aade (127.31.131.67:33619)
I20240507 02:09:41.301072 13152 raft_consensus.cc:2781] T dbb194f09b9f49298ea4cdb86f13dbb8 P 78ed6a7374fc41cd8f80970532ee24cb [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:09:41.301146 13152 raft_consensus.cc:686] T dbb194f09b9f49298ea4cdb86f13dbb8 P 78ed6a7374fc41cd8f80970532ee24cb [term 1 LEADER]: Becoming Leader. State: Replica: 78ed6a7374fc41cd8f80970532ee24cb, State: Running, Role: LEADER
I20240507 02:09:41.301259 13152 consensus_queue.cc:237] T dbb194f09b9f49298ea4cdb86f13dbb8 P 78ed6a7374fc41cd8f80970532ee24cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "322a5d3259b84784a9ab75d487df2513" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35639 } } peers { permanent_uuid: "78ed6a7374fc41cd8f80970532ee24cb" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 39789 } } peers { permanent_uuid: "7c6b66e4a60041ecbd1811f7be53aade" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 33619 } }
I20240507 02:09:41.302090 12644 catalog_manager.cc:5484] T dbb194f09b9f49298ea4cdb86f13dbb8 P 78ed6a7374fc41cd8f80970532ee24cb reported cstate change: term changed from 0 to 1, leader changed from <none> to 78ed6a7374fc41cd8f80970532ee24cb (127.31.131.66). New cstate: current_term: 1 leader_uuid: "78ed6a7374fc41cd8f80970532ee24cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "322a5d3259b84784a9ab75d487df2513" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78ed6a7374fc41cd8f80970532ee24cb" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 39789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c6b66e4a60041ecbd1811f7be53aade" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 33619 } health_report { overall_health: UNKNOWN } } }
I20240507 02:09:41.320021 13140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:41.317201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42156 (local address 127.31.131.66:39789)
0507 02:09:41.317253 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0507 02:09:41.317254 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:09:41.317261 (+ 7us) server_negotiation.cc:407] Connection header received
0507 02:09:41.317293 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:09:41.317294 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:09:41.317310 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:09:41.317369 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0507 02:09:41.317708 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:41.318447 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:09:41.319344 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:41.319520 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:41.319557 (+ 37us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0507 02:09:41.319660 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0507 02:09:41.319736 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0507 02:09:41.319929 (+ 193us) server_negotiation.cc:299] Negotiation successful
0507 02:09:41.319949 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20240507 02:09:41.321928 13065 raft_consensus.cc:1261] T dbb194f09b9f49298ea4cdb86f13dbb8 P 7c6b66e4a60041ecbd1811f7be53aade [term 1 FOLLOWER]: Refusing update from remote peer 78ed6a7374fc41cd8f80970532ee24cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 02:09:41.322247 13155 consensus_queue.cc:1035] T dbb194f09b9f49298ea4cdb86f13dbb8 P 78ed6a7374fc41cd8f80970532ee24cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c6b66e4a60041ecbd1811f7be53aade" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 33619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240507 02:09:41.323951 12844 tablet.cc:2369] T dbb194f09b9f49298ea4cdb86f13dbb8 P 322a5d3259b84784a9ab75d487df2513: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240507 02:09:41.324131 12797 raft_consensus.cc:1261] T dbb194f09b9f49298ea4cdb86f13dbb8 P 322a5d3259b84784a9ab75d487df2513 [term 1 FOLLOWER]: Refusing update from remote peer 78ed6a7374fc41cd8f80970532ee24cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 02:09:41.324426 13155 consensus_queue.cc:1035] T dbb194f09b9f49298ea4cdb86f13dbb8 P 78ed6a7374fc41cd8f80970532ee24cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "322a5d3259b84784a9ab75d487df2513" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 02:09:41.327571 13158 mvcc.cc:205] Tried to move back new op lower bound from 7024835712292216832 to 7024835712210771968. Current Snapshot: MvccSnapshot[applied={T|T < 7024835712292216832}]
I20240507 02:09:41.328064 13161 mvcc.cc:205] Tried to move back new op lower bound from 7024835712292216832 to 7024835712210771968. Current Snapshot: MvccSnapshot[applied={T|T < 7024835712292216832}]
I20240507 02:09:41.330070 13159 mvcc.cc:205] Tried to move back new op lower bound from 7024835712292216832 to 7024835712210771968. Current Snapshot: MvccSnapshot[applied={T|T < 7024835712292216832}]
I20240507 02:09:41.332070 12644 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53092:
table { table_name: "test-table" } modify_external_catalogs: true
I20240507 02:09:41.332151 12644 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53092:
table { table_name: "test-table" } modify_external_catalogs: true
I20240507 02:09:41.332860 12644 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 902d0b3c51a44c65b0d2f7e6ffe9a301: Sending DeleteTablet for 3 replicas of tablet dbb194f09b9f49298ea4cdb86f13dbb8
I20240507 02:09:41.333285 12777 tablet_service.cc:1514] Processing DeleteTablet for tablet dbb194f09b9f49298ea4cdb86f13dbb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:09:41 UTC) from {username='kudu', principal='kudu/127.31.131.126@KRBTEST.COM'} at 127.0.0.1:41506
I20240507 02:09:41.333478 12900 tablet_service.cc:1514] Processing DeleteTablet for tablet dbb194f09b9f49298ea4cdb86f13dbb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:09:41 UTC) from {username='kudu', principal='kudu/127.31.131.126@KRBTEST.COM'} at 127.0.0.1:42150
I20240507 02:09:41.333642 13027 tablet_service.cc:1514] Processing DeleteTablet for tablet dbb194f09b9f49298ea4cdb86f13dbb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:09:41 UTC) from {username='kudu', principal='kudu/127.31.131.126@KRBTEST.COM'} at 127.0.0.1:45036
I20240507 02:09:41.334296 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 12683
I20240507 02:09:41.335777 13168 tablet_replica.cc:330] T dbb194f09b9f49298ea4cdb86f13dbb8 P 78ed6a7374fc41cd8f80970532ee24cb: stopping tablet replica
I20240507 02:09:41.335893 13169 tablet_replica.cc:330] T dbb194f09b9f49298ea4cdb86f13dbb8 P 7c6b66e4a60041ecbd1811f7be53aade: stopping tablet replica
I20240507 02:09:41.336050 13169 raft_consensus.cc:2229] T dbb194f09b9f49298ea4cdb86f13dbb8 P 7c6b66e4a60041ecbd1811f7be53aade [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 02:09:41.336149 13168 raft_consensus.cc:2229] T dbb194f09b9f49298ea4cdb86f13dbb8 P 78ed6a7374fc41cd8f80970532ee24cb [term 1 LEADER]: Raft consensus shutting down.
I20240507 02:09:41.336285 13168 raft_consensus.cc:2258] T dbb194f09b9f49298ea4cdb86f13dbb8 P 78ed6a7374fc41cd8f80970532ee24cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 02:09:41.336377 13169 raft_consensus.cc:2258] T dbb194f09b9f49298ea4cdb86f13dbb8 P 7c6b66e4a60041ecbd1811f7be53aade [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 02:09:41.336828 13168 ts_tablet_manager.cc:1875] T dbb194f09b9f49298ea4cdb86f13dbb8 P 78ed6a7374fc41cd8f80970532ee24cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 02:09:41.339466 13168 ts_tablet_manager.cc:1888] T dbb194f09b9f49298ea4cdb86f13dbb8 P 78ed6a7374fc41cd8f80970532ee24cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240507 02:09:41.339982 13169 ts_tablet_manager.cc:1875] T dbb194f09b9f49298ea4cdb86f13dbb8 P 7c6b66e4a60041ecbd1811f7be53aade: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240507 02:09:41.341168 12631 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.131.65:35639 (error 108)
I20240507 02:09:41.341496 13169 ts_tablet_manager.cc:1888] T dbb194f09b9f49298ea4cdb86f13dbb8 P 7c6b66e4a60041ecbd1811f7be53aade: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240507 02:09:41.341606 13169 log.cc:1196] T dbb194f09b9f49298ea4cdb86f13dbb8 P 7c6b66e4a60041ecbd1811f7be53aade: Deleting WAL directory at /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1715047604443657-32269-0/minicluster-data/ts-2/wal/wals/dbb194f09b9f49298ea4cdb86f13dbb8
I20240507 02:09:41.341719 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 12847
I20240507 02:09:41.343179 13169 ts_tablet_manager.cc:1909] T dbb194f09b9f49298ea4cdb86f13dbb8 P 7c6b66e4a60041ecbd1811f7be53aade: Deleting consensus metadata
I20240507 02:09:41.343619 12630 catalog_manager.cc:4830] TS 7c6b66e4a60041ecbd1811f7be53aade (127.31.131.67:33619): tablet dbb194f09b9f49298ea4cdb86f13dbb8 (table test-table [id=a5d56ef02bba41bd938ef261be808962]) successfully deleted
I20240507 02:09:41.348307 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 12980
I20240507 02:09:41.349751 13141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:41.348616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.65:35639 (local address 127.0.0.1:41512)
0507 02:09:41.349661 (+ 1045us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:41.349709 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.65:35639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1013}
W20240507 02:09:41.351626 12631 catalog_manager.cc:4573] TS 322a5d3259b84784a9ab75d487df2513 (127.31.131.65:35639): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dbb194f09b9f49298ea4cdb86f13dbb8: Network error: Client connection negotiation failed: client connection to 127.31.131.65:35639: connect: Connection refused (error 111)
I20240507 02:09:41.351966 13141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:41.351873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.66:39789 (local address 127.0.0.1:42170)
0507 02:09:41.351923 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:41.351942 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.66:39789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20240507 02:09:41.355569 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 12612
I20240507 02:09:41.361789 32269 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20240507 02:09:41.699220 32269 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20240507 02:09:41.699317 32269 mini_ranger.cc:66] Stopping Ranger...
I20240507 02:09:42.032404 32269 mini_ranger.cc:68] Stopped Ranger
2024-05-07 02:09:42.032 UTC [11679] LOG: received smart shutdown request
2024-05-07 02:09:42.035 UTC [11679] LOG: background worker "logical replication launcher" (PID 11688) exited with exit code 1
2024-05-07 02:09:42.036 UTC [11683] LOG: shutting down
2024-05-07 02:09:42.170 UTC [11679] LOG: database system is shut down
2024-05-07T02:09:42Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (43557 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715047604443657-32269-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 07 02:09:42 dist-test-slave-v0bg krb5kdc[13179](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 07 02:09:42 dist-test-slave-v0bg krb5kdc[13179](info): set up 2 sockets
May 07 02:09:42 dist-test-slave-v0bg krb5kdc[13179](info): commencing operation
krb5kdc: starting...
W20240507 02:09:43.143226 32269 mini_kdc.cc:121] Time spent starting KDC: real 0.870s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715047604443657-32269-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715047604443657-32269-0/krb5kdc/test-user.keytab.
May 07 02:09:43 dist-test-slave-v0bg krb5kdc[13179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-07T02:09:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-07T02:09:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.131.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab.
Entry for principal kudu/127.31.131.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab.
I20240507 02:09:43.240629 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715047604443657-32269-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715047604443657-32269-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715047604443657-32269-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:36097
--webserver_interface=127.31.131.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab
--principal=kudu/127.31.131.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:35209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.131.126:36097,127.31.131.125:42347,127.31.131.124:36111
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:09:43.314297 13192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:09:43.314464 13192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:09:43.314484 13192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:09:43.315703 13192 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240507 02:09:43.315743 13192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:09:43.315755 13192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:09:43.315765 13192 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240507 02:09:43.315775 13192 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240507 02:09:43.317240 13192 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:35209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715047604443657-32269-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715047604443657-32269-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.131.126:36097,127.31.131.125:42347,127.31.131.124:36111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab
--principal=kudu/127.31.131.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:36097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.131.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715047604443657-32269-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:09:43.317385 13192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:09:43.317516 13192 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:09:43.320765 13197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:09:43.320799 13192 server_base.cc:1034] running on GCE node
W20240507 02:09:43.320920 13198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:09:43.321049 13200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:09:43.321244 13192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:09:43.321478 13192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:09:43.322587 13192 hybrid_clock.cc:648] HybridClock initialized: now 1715047783322576 us; error 29 us; skew 500 ppm
May 07 02:09:43 dist-test-slave-v0bg krb5kdc[13179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047783, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:09:43.328454 13192 init.cc:377] Logged in from keytab as kudu/127.31.131.126@KRBTEST.COM (short username kudu)
I20240507 02:09:43.328801 13192 webserver.cc:460] Webserver started at http://127.31.131.126:40191/ using document root <none> and password file <none>
I20240507 02:09:43.328944 13192 fs_manager.cc:359] Metadata directory not provided
I20240507 02:09:43.328982 13192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:09:43.329074 13192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:09:43.329928 13192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715047604443657-32269-0/minicluster-data/master-0/data/instance:
uuid: "c4cf873b2b65476eb38261d37c390ad7"
format_stamp: "Formatted at 2024-05-07 02:09:43 on dist-test-slave-v0bg"
server_key: "c5abb3d1ec4e3bf88c4a7303833c8cfa"
server_key_iv: "02b63416a49d5b79f29b77369404de87"
server_key_version: "encryptionkey@0"
I20240507 02:09:43.330185 13192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715047604443657-32269-0/minicluster-data/master-0/wal/instance:
uuid: "c4cf873b2b65476eb38261d37c390ad7"
format_stamp: "Formatted at 2024-05-07 02:09:43 on dist-test-slave-v0bg"
server_key: "c5abb3d1ec4e3bf88c4a7303833c8cfa"
server_key_iv: "02b63416a49d5b79f29b77369404de87"
server_key_version: "encryptionkey@0"
I20240507 02:09:43.331215 13192 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240507 02:09:43.331758 13207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:09:43.331897 13192 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240507 02:09:43.331964 13192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715047604443657-32269-0/minicluster-data/master-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715047604443657-32269-0/minicluster-data/master-0/wal
uuid: "c4cf873b2b65476eb38261d37c390ad7"
format_stamp: "Formatted at 2024-05-07 02:09:43 on dist-test-slave-v0bg"
server_key: "c5abb3d1ec4e3bf88c4a7303833c8cfa"
server_key_iv: "02b63416a49d5b79f29b77369404de87"
server_key_version: "encryptionkey@0"
I20240507 02:09:43.332015 13192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715047604443657-32269-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715047604443657-32269-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715047604443657-32269-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:09:43.345147 13192 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:09:43.346526 13192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:09:43.346647 13192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:09:43.350160 13192 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.126:36097
I20240507 02:09:43.350200 13259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.126:36097 every 8 connection(s)
I20240507 02:09:43.350432 13192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
I20240507 02:09:43.350946 13260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:09:43.352461 13260 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.131.126" port: 36097 } has no permanent_uuid. Determining permanent_uuid...
I20240507 02:09:43.353744 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 13192
I20240507 02:09:43.353832 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715047604443657-32269-0/minicluster-data/master-0/wal/instance
I20240507 02:09:43.353982 32269 external_mini_cluster.cc:1428] Setting key ef8199fbc66411d2a6605929a916a6d0
May 07 02:09:43 dist-test-slave-v0bg krb5kdc[13179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047783, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.126@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.131.125@KRBTEST.COM; defaulting to no policy
I20240507 02:09:43.359673 13263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:43.353073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60758 (local address 127.31.131.126:36097)
0507 02:09:43.353168 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0507 02:09:43.353176 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0507 02:09:43.353194 (+ 18us) server_negotiation.cc:407] Connection header received
0507 02:09:43.353631 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:09:43.353633 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:09:43.353647 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:09:43.353714 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:09:43.354336 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:43.355027 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:09:43.355488 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:43.355655 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:43.357249 (+ 1594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:09:43.357256 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:09:43.357257 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:09:43.357261 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:09:43.358583 (+ 1322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:09:43.358866 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:09:43.358868 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:09:43.358869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:09:43.358901 (+ 32us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:09:43.359045 (+ 144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:09:43.359046 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:09:43.359046 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:09:43.359183 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:09:43.359248 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0507 02:09:43.359401 (+ 153us) server_negotiation.cc:299] Negotiation successful
0507 02:09:43.359568 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":87,"server-negotiator.queue_time_us":75,"thread_start_us":44,"threads_started":1}
I20240507 02:09:43.359673 13262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:43.352953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:36097 (local address 127.0.0.1:60758)
0507 02:09:43.353076 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:43.353098 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0507 02:09:43.353509 (+ 411us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:09:43.353724 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:09:43.353726 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:09:43.354030 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:09:43.354258 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:43.354262 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:43.355090 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:09:43.355091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:09:43.355424 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:43.355425 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:43.355533 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:43.355982 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:09:43.355995 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:09:43.357177 (+ 1182us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:09:43.358655 (+ 1478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:09:43.358659 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:09:43.358661 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:09:43.358806 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:09:43.358949 (+ 143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:09:43.358950 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:09:43.358950 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:09:43.358998 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:09:43.359233 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:09:43.359237 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:09:43.359295 (+ 58us) client_negotiation.cc:769] Sending connection context
0507 02:09:43.359374 (+ 79us) client_negotiation.cc:240] Negotiation successful
0507 02:09:43.359495 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":56,"threads_started":1}
I20240507 02:09:43.360282 13260 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.131.125" port: 42347 } has no permanent_uuid. Determining permanent_uuid...
I20240507 02:09:43.360536 13262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:43.360444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.125:42347 (local address 127.0.0.1:53282)
0507 02:09:43.360482 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:43.360519 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.125:42347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20240507 02:09:43.360635 13209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.131.125:42347: connect: Connection refused (error 111)
I20240507 02:09:43.360960 13262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:43.360897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.125:42347 (local address 127.0.0.1:53296)
0507 02:09:43.360925 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:43.360946 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.125:42347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20240507 02:09:43.361032 13260 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.31.131.125:42347: Network error: Client connection negotiation failed: client connection to 127.31.131.125:42347: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.125.keytab.
Entry for principal kudu/127.31.131.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.125.keytab.
I20240507 02:09:43.372910 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715047604443657-32269-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715047604443657-32269-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715047604443657-32269-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715047604443657-32269-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.125:42347
--webserver_interface=127.31.131.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.125.keytab
--principal=kudu/127.31.131.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:35209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.131.126:36097,127.31.131.125:42347,127.31.131.124:36111
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240507 02:09:43.410188 13260 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.131.125" port: 42347 } attempt: 1
I20240507 02:09:43.410573 13262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:43.410453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.125:42347 (local address 127.0.0.1:53304)
0507 02:09:43.410503 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:43.410528 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.125:42347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20240507 02:09:43.410884 13262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:43.410801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.125:42347 (local address 127.0.0.1:53314)
0507 02:09:43.410840 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:43.410862 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.125:42347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20240507 02:09:43.410972 13260 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.31.131.125:42347: Network error: Client connection negotiation failed: client connection to 127.31.131.125:42347: connect: Connection refused (error 111)
W20240507 02:09:43.445459 13267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:09:43.445612 13267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:09:43.445644 13267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:09:43.446905 13267 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240507 02:09:43.446942 13267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:09:43.446954 13267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:09:43.446977 13267 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240507 02:09:43.447000 13267 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240507 02:09:43.448508 13267 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:35209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715047604443657-32269-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715047604443657-32269-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.31.131.126:36097,127.31.131.125:42347,127.31.131.124:36111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.125.keytab
--principal=kudu/127.31.131.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.125:42347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715047604443657-32269-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.31.131.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13267
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715047604443657-32269-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:09:43.448678 13267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:09:43.448834 13267 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:09:43.452033 13272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:09:43.452065 13273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:09:43.452039 13275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:09:43.452244 13267 server_base.cc:1034] running on GCE node
I20240507 02:09:43.452395 13267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:09:43.452555 13267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:09:43.453676 13267 hybrid_clock.cc:648] HybridClock initialized: now 1715047783453668 us; error 27 us; skew 500 ppm
May 07 02:09:43 dist-test-slave-v0bg krb5kdc[13179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047783, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:09:43.455737 13267 init.cc:377] Logged in from keytab as kudu/127.31.131.125@KRBTEST.COM (short username kudu)
I20240507 02:09:43.456079 13267 webserver.cc:460] Webserver started at http://127.31.131.125:42907/ using document root <none> and password file <none>
I20240507 02:09:43.456207 13267 fs_manager.cc:359] Metadata directory not provided
I20240507 02:09:43.456238 13267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:09:43.456308 13267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:09:43.457104 13267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715047604443657-32269-0/minicluster-data/master-1/data/instance:
uuid: "629a4752d61d4b21be97a73698100499"
format_stamp: "Formatted at 2024-05-07 02:09:43 on dist-test-slave-v0bg"
server_key: "77ea13459436527222af2a24064f548a"
server_key_iv: "5c0268915fa0a865e8b0814188662a2a"
server_key_version: "encryptionkey@0"
I20240507 02:09:43.457329 13267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715047604443657-32269-0/minicluster-data/master-1/wal/instance:
uuid: "629a4752d61d4b21be97a73698100499"
format_stamp: "Formatted at 2024-05-07 02:09:43 on dist-test-slave-v0bg"
server_key: "77ea13459436527222af2a24064f548a"
server_key_iv: "5c0268915fa0a865e8b0814188662a2a"
server_key_version: "encryptionkey@0"
I20240507 02:09:43.458248 13267 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240507 02:09:43.458806 13282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:09:43.458940 13267 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240507 02:09:43.459002 13267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715047604443657-32269-0/minicluster-data/master-1/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715047604443657-32269-0/minicluster-data/master-1/wal
uuid: "629a4752d61d4b21be97a73698100499"
format_stamp: "Formatted at 2024-05-07 02:09:43 on dist-test-slave-v0bg"
server_key: "77ea13459436527222af2a24064f548a"
server_key_iv: "5c0268915fa0a865e8b0814188662a2a"
server_key_version: "encryptionkey@0"
I20240507 02:09:43.459055 13267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715047604443657-32269-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715047604443657-32269-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715047604443657-32269-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:09:43.471868 13267 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:09:43.473608 13267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:09:43.473706 13267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:09:43.477191 13267 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.125:42347
I20240507 02:09:43.477218 13334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.125:42347 every 8 connection(s)
I20240507 02:09:43.477452 13267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715047604443657-32269-0/minicluster-data/master-1/data/info.pb
I20240507 02:09:43.477917 13335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:09:43.479111 13260 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.131.125" port: 42347 } attempt: 2
I20240507 02:09:43.479544 13335 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.131.126" port: 36097 } has no permanent_uuid. Determining permanent_uuid...
May 07 02:09:43 dist-test-slave-v0bg krb5kdc[13179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047783, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.126@KRBTEST.COM for kudu/127.31.131.125@KRBTEST.COM
May 07 02:09:43 dist-test-slave-v0bg krb5kdc[13179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047783, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.125@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:09:43.485713 13262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:43.479369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.125:42347 (local address 127.0.0.1:53330)
0507 02:09:43.479413 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:43.479423 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0507 02:09:43.479466 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:09:43.480305 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:09:43.480306 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:09:43.480455 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:09:43.480596 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:43.480597 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:43.481370 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:09:43.481371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:09:43.481701 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:43.481702 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:43.481757 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:43.482236 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:09:43.482241 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:09:43.483336 (+ 1095us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:09:43.484844 (+ 1508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:09:43.484844 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:09:43.484845 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:09:43.485003 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:09:43.485151 (+ 148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:09:43.485152 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:09:43.485153 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:09:43.485180 (+ 27us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:09:43.485436 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:09:43.485437 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:09:43.485476 (+ 39us) client_negotiation.cc:769] Sending connection context
0507 02:09:43.485539 (+ 63us) client_negotiation.cc:240] Negotiation successful
0507 02:09:43.485624 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20240507 02:09:43.485842 13337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:43.479520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53330 (local address 127.31.131.125:42347)
0507 02:09:43.479645 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0507 02:09:43.479653 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0507 02:09:43.479668 (+ 15us) server_negotiation.cc:407] Connection header received
0507 02:09:43.480155 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:09:43.480162 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:09:43.480203 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:09:43.480293 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:09:43.480668 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:43.481306 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:09:43.481767 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:43.481941 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:43.483414 (+ 1473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:09:43.483422 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:09:43.483423 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:09:43.483428 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:09:43.484768 (+ 1340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:09:43.485065 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:09:43.485068 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:09:43.485068 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:09:43.485101 (+ 33us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:09:43.485232 (+ 131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:09:43.485233 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:09:43.485233 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:09:43.485381 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:09:43.485447 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0507 02:09:43.485598 (+ 151us) server_negotiation.cc:299] Negotiation successful
0507 02:09:43.485688 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":56,"threads_started":1}
I20240507 02:09:43.486125 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 13267
I20240507 02:09:43.486212 13263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:43.480023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60772 (local address 127.31.131.126:36097)
0507 02:09:43.480069 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0507 02:09:43.480071 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:09:43.480192 (+ 121us) server_negotiation.cc:407] Connection header received
0507 02:09:43.480280 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:09:43.480281 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:09:43.480290 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:09:43.480341 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:09:43.480912 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:43.481335 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:09:43.481788 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:43.481945 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:43.483989 (+ 2044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:09:43.483992 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:09:43.483993 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:09:43.483997 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:09:43.485197 (+ 1200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:09:43.485483 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:09:43.485484 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:09:43.485484 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:09:43.485503 (+ 19us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:09:43.485618 (+ 115us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:09:43.485619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:09:43.485619 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:09:43.485690 (+ 71us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:09:43.485739 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0507 02:09:43.486066 (+ 327us) server_negotiation.cc:299] Negotiation successful
0507 02:09:43.486133 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20240507 02:09:43.486220 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715047604443657-32269-0/minicluster-data/master-1/wal/instance
I20240507 02:09:43.486390 32269 external_mini_cluster.cc:1428] Setting key 5dc0396fbe1c78580885000e2c657ea0
I20240507 02:09:43.486441 13260 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.131.124" port: 36111 } has no permanent_uuid. Determining permanent_uuid...
I20240507 02:09:43.486657 13338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:43.479982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:36097 (local address 127.0.0.1:60772)
0507 02:09:43.480110 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:43.480131 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0507 02:09:43.480203 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:09:43.480357 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:09:43.480359 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:09:43.480620 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:09:43.480832 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:43.480835 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:43.481398 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:09:43.481399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:09:43.481724 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:43.481725 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:43.481835 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:43.482301 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:09:43.482314 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:09:43.483919 (+ 1605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:09:43.485274 (+ 1355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:09:43.485279 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:09:43.485281 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:09:43.485424 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:09:43.485551 (+ 127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:09:43.485552 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:09:43.485553 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:09:43.485581 (+ 28us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:09:43.485737 (+ 156us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:09:43.485741 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:09:43.485784 (+ 43us) client_negotiation.cc:769] Sending connection context
0507 02:09:43.485888 (+ 104us) client_negotiation.cc:240] Negotiation successful
0507 02:09:43.485971 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":67,"threads_started":1}
I20240507 02:09:43.486719 13262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:43.486636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.124:36111 (local address 127.0.0.1:33350)
0507 02:09:43.486673 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:43.486697 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.124:36111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20240507 02:09:43.487040 13262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:43.486951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.124:36111 (local address 127.0.0.1:33360)
0507 02:09:43.486991 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:43.487020 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.124:36111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20240507 02:09:43.487097 13335 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.131.125" port: 42347 } has no permanent_uuid. Determining permanent_uuid...
W20240507 02:09:43.487119 13260 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.31.131.124:36111: Network error: Client connection negotiation failed: client connection to 127.31.131.124:36111: connect: Connection refused (error 111)
May 07 02:09:43 dist-test-slave-v0bg krb5kdc[13179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047783, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.125@KRBTEST.COM for kudu/127.31.131.125@KRBTEST.COM
I20240507 02:09:43.492236 13337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:43.487318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53346 (local address 127.31.131.125:42347)
0507 02:09:43.487346 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0507 02:09:43.487348 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:09:43.487355 (+ 7us) server_negotiation.cc:407] Connection header received
0507 02:09:43.487413 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:09:43.487414 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:09:43.487420 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:09:43.487452 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:09:43.487818 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:43.488392 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:09:43.488823 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:43.488980 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:43.490344 (+ 1364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:09:43.490346 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:09:43.490347 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:09:43.490351 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:09:43.491445 (+ 1094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:09:43.491769 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:09:43.491770 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:09:43.491770 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:09:43.491784 (+ 14us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:09:43.491908 (+ 124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:09:43.491908 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:09:43.491908 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:09:43.491968 (+ 60us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:09:43.492005 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0507 02:09:43.492132 (+ 127us) server_negotiation.cc:299] Negotiation successful
0507 02:09:43.492187 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20240507 02:09:43.492352 13338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:43.487291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.125:42347 (local address 127.0.0.1:53346)
0507 02:09:43.487325 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:43.487332 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0507 02:09:43.487361 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:09:43.487456 (+ 95us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:09:43.487456 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:09:43.487579 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:09:43.487745 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:43.487746 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:43.488459 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:09:43.488460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:09:43.488764 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:43.488765 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:43.488806 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:43.489121 (+ 315us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:09:43.489125 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:09:43.490272 (+ 1147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:09:43.491519 (+ 1247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:09:43.491520 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:09:43.491520 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:09:43.491677 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:09:43.491834 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:09:43.491834 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:09:43.491835 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:09:43.491862 (+ 27us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:09:43.492014 (+ 152us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:09:43.492014 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:09:43.492053 (+ 39us) client_negotiation.cc:769] Sending connection context
0507 02:09:43.492121 (+ 68us) client_negotiation.cc:240] Negotiation successful
0507 02:09:43.492249 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24,"mutex_wait_us":43}
I20240507 02:09:43.492691 13335 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.131.124" port: 36111 } has no permanent_uuid. Determining permanent_uuid...
I20240507 02:09:43.492947 13338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:43.492852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.124:36111 (local address 127.0.0.1:33376)
0507 02:09:43.492882 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:43.492925 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.124:36111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
WARNING: no policy specified for kudu/127.31.131.124@KRBTEST.COM; defaulting to no policy
W20240507 02:09:43.493037 13285 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.131.124:36111: connect: Connection refused (error 111)
I20240507 02:09:43.493366 13338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:43.493288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.124:36111 (local address 127.0.0.1:33388)
0507 02:09:43.493326 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:43.493348 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.124:36111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20240507 02:09:43.493505 13335 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.31.131.124:36111: Network error: Client connection negotiation failed: client connection to 127.31.131.124:36111: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.124@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.124.keytab.
Entry for principal kudu/127.31.131.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.124.keytab.
I20240507 02:09:43.506657 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715047604443657-32269-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715047604443657-32269-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715047604443657-32269-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715047604443657-32269-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.124:36111
--webserver_interface=127.31.131.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.124.keytab
--principal=kudu/127.31.131.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:35209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.131.126:36097,127.31.131.125:42347,127.31.131.124:36111
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240507 02:09:43.530256 13260 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.131.124" port: 36111 } attempt: 1
I20240507 02:09:43.530627 13262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:43.530498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.124:36111 (local address 127.0.0.1:33392)
0507 02:09:43.530564 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:43.530599 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.124:36111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20240507 02:09:43.530925 13262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:43.530862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.124:36111 (local address 127.0.0.1:33396)
0507 02:09:43.530892 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:43.530910 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.124:36111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20240507 02:09:43.531005 13260 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.31.131.124:36111: Network error: Client connection negotiation failed: client connection to 127.31.131.124:36111: connect: Connection refused (error 111)
I20240507 02:09:43.542678 13335 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.131.124" port: 36111 } attempt: 1
I20240507 02:09:43.542980 13338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:43.542891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.124:36111 (local address 127.0.0.1:33410)
0507 02:09:43.542937 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:43.542959 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.124:36111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20240507 02:09:43.543246 13338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:43.543179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.124:36111 (local address 127.0.0.1:33422)
0507 02:09:43.543211 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:43.543231 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.124:36111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20240507 02:09:43.543329 13335 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.31.131.124:36111: Network error: Client connection negotiation failed: client connection to 127.31.131.124:36111: connect: Connection refused (error 111)
I20240507 02:09:43.578176 13260 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.131.124" port: 36111 } attempt: 2
I20240507 02:09:43.578537 13262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:43.578437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.124:36111 (local address 127.0.0.1:33432)
0507 02:09:43.578487 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:43.578510 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.124:36111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20240507 02:09:43.578884 13262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:43.578806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.124:36111 (local address 127.0.0.1:33434)
0507 02:09:43.578848 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:43.578869 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.124:36111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20240507 02:09:43.578963 13260 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.31.131.124:36111: Network error: Client connection negotiation failed: client connection to 127.31.131.124:36111: connect: Connection refused (error 111)
W20240507 02:09:43.582440 13342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:09:43.582590 13342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:09:43.582629 13342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:09:43.583847 13342 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240507 02:09:43.583884 13342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:09:43.583895 13342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:09:43.583918 13342 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240507 02:09:43.583940 13342 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240507 02:09:43.585438 13342 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:35209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715047604443657-32269-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715047604443657-32269-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.31.131.126:36097,127.31.131.125:42347,127.31.131.124:36111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.124.keytab
--principal=kudu/127.31.131.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.124:36111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715047604443657-32269-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.31.131.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715047604443657-32269-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:09:43.585605 13342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:09:43.585764 13342 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:09:43.588869 13347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:09:43.588877 13348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:09:43.589001 13350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:09:43.589095 13342 server_base.cc:1034] running on GCE node
I20240507 02:09:43.589272 13342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:09:43.589488 13342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:09:43.590592 13342 hybrid_clock.cc:648] HybridClock initialized: now 1715047783590579 us; error 25 us; skew 500 ppm
May 07 02:09:43 dist-test-slave-v0bg krb5kdc[13179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047783, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:09:43.592527 13342 init.cc:377] Logged in from keytab as kudu/127.31.131.124@KRBTEST.COM (short username kudu)
I20240507 02:09:43.592864 13342 webserver.cc:460] Webserver started at http://127.31.131.124:42209/ using document root <none> and password file <none>
I20240507 02:09:43.592994 13342 fs_manager.cc:359] Metadata directory not provided
I20240507 02:09:43.593025 13342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:09:43.593097 13342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:09:43.593899 13342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715047604443657-32269-0/minicluster-data/master-2/data/instance:
uuid: "b81b4b0655dd46e690e9120e0f2ef20e"
format_stamp: "Formatted at 2024-05-07 02:09:43 on dist-test-slave-v0bg"
server_key: "42e6d5ed9b55b34de2c4cad09cc5b696"
server_key_iv: "2dcf6d019939ce517f8964a67a1b5bd5"
server_key_version: "encryptionkey@0"
I20240507 02:09:43.594122 13342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715047604443657-32269-0/minicluster-data/master-2/wal/instance:
uuid: "b81b4b0655dd46e690e9120e0f2ef20e"
format_stamp: "Formatted at 2024-05-07 02:09:43 on dist-test-slave-v0bg"
server_key: "42e6d5ed9b55b34de2c4cad09cc5b696"
server_key_iv: "2dcf6d019939ce517f8964a67a1b5bd5"
server_key_version: "encryptionkey@0"
I20240507 02:09:43.595129 13342 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240507 02:09:43.595665 13357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:09:43.595824 13342 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240507 02:09:43.595883 13342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715047604443657-32269-0/minicluster-data/master-2/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715047604443657-32269-0/minicluster-data/master-2/wal
uuid: "b81b4b0655dd46e690e9120e0f2ef20e"
format_stamp: "Formatted at 2024-05-07 02:09:43 on dist-test-slave-v0bg"
server_key: "42e6d5ed9b55b34de2c4cad09cc5b696"
server_key_iv: "2dcf6d019939ce517f8964a67a1b5bd5"
server_key_version: "encryptionkey@0"
I20240507 02:09:43.595927 13342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715047604443657-32269-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715047604443657-32269-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715047604443657-32269-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:09:43.611456 13335 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.131.124" port: 36111 } attempt: 2
I20240507 02:09:43.611766 13338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:43.611661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.124:36111 (local address 127.0.0.1:33436)
0507 02:09:43.611709 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:43.611740 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.124:36111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20240507 02:09:43.612017 13338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:43.611956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.124:36111 (local address 127.0.0.1:33442)
0507 02:09:43.611985 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:43.612003 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.124:36111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20240507 02:09:43.612083 13335 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.31.131.124:36111: Network error: Client connection negotiation failed: client connection to 127.31.131.124:36111: connect: Connection refused (error 111)
I20240507 02:09:43.616860 13342 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:09:43.618602 13342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:09:43.618685 13342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:09:43.621896 13342 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.124:36111
I20240507 02:09:43.621937 13409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.124:36111 every 8 connection(s)
I20240507 02:09:43.622164 13342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715047604443657-32269-0/minicluster-data/master-2/data/info.pb
I20240507 02:09:43.622670 13410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:09:43.624182 13410 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.131.126" port: 36097 } has no permanent_uuid. Determining permanent_uuid...
May 07 02:09:43 dist-test-slave-v0bg krb5kdc[13179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047783, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.124@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:09:43.629923 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 13342
I20240507 02:09:43.630023 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACert.1715047604443657-32269-0/minicluster-data/master-2/wal/instance
I20240507 02:09:43.630198 32269 external_mini_cluster.cc:1428] Setting key 68ccffc7b17f9967c8eee0fab6ef9cbc
I20240507 02:09:43.630475 13263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:43.624691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60784 (local address 127.31.131.126:36097)
0507 02:09:43.624741 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0507 02:09:43.624743 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:09:43.624882 (+ 139us) server_negotiation.cc:407] Connection header received
0507 02:09:43.625276 (+ 394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:09:43.625277 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:09:43.625285 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:09:43.625324 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:09:43.625919 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:43.626334 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:09:43.626839 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:43.626946 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:43.628427 (+ 1481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:09:43.628430 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:09:43.628430 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:09:43.628435 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:09:43.629555 (+ 1120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:09:43.629834 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:09:43.629835 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:09:43.629836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:09:43.629856 (+ 20us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:09:43.629981 (+ 125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:09:43.629982 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:09:43.629982 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:09:43.630083 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:09:43.630137 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0507 02:09:43.630345 (+ 208us) server_negotiation.cc:299] Negotiation successful
0507 02:09:43.630405 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20240507 02:09:43.630620 13412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:43.624658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:36097 (local address 127.0.0.1:60784)
0507 02:09:43.624792 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:43.624817 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0507 02:09:43.625186 (+ 369us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:09:43.625372 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:09:43.625374 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:09:43.625602 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:09:43.625859 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:43.625862 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:43.626394 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:09:43.626394 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:09:43.626778 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:43.626780 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:43.626871 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:43.627218 (+ 347us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:09:43.627229 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:09:43.628354 (+ 1125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:09:43.629611 (+ 1257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:09:43.629613 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:09:43.629615 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:09:43.629783 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:09:43.629906 (+ 123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:09:43.629906 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:09:43.629906 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:09:43.629947 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:09:43.630135 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:09:43.630139 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:09:43.630231 (+ 92us) client_negotiation.cc:769] Sending connection context
0507 02:09:43.630323 (+ 92us) client_negotiation.cc:240] Negotiation successful
0507 02:09:43.630436 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":68,"threads_started":1}
I20240507 02:09:43.631201 13410 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.131.125" port: 42347 } has no permanent_uuid. Determining permanent_uuid...
May 07 02:09:43 dist-test-slave-v0bg krb5kdc[13179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047783, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.124@KRBTEST.COM for kudu/127.31.131.125@KRBTEST.COM
May 07 02:09:43 dist-test-slave-v0bg krb5kdc[13179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.131.124@KRBTEST.COM
I20240507 02:09:43.637037 13337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:43.631459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53356 (local address 127.31.131.125:42347)
0507 02:09:43.631495 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0507 02:09:43.631497 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:09:43.631536 (+ 39us) server_negotiation.cc:407] Connection header received
0507 02:09:43.631600 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:09:43.631600 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:09:43.631609 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:09:43.631656 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:09:43.631973 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:43.632607 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:09:43.633227 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:43.633731 (+ 504us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:43.634888 (+ 1157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:09:43.634891 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:09:43.634892 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:09:43.634898 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:09:43.636195 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:09:43.636529 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:09:43.636530 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:09:43.636530 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:09:43.636548 (+ 18us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:09:43.636671 (+ 123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:09:43.636672 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:09:43.636672 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:09:43.636740 (+ 68us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:09:43.636787 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0507 02:09:43.636931 (+ 144us) server_negotiation.cc:299] Negotiation successful
0507 02:09:43.636982 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20240507 02:09:43.637089 13412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:43.631407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.125:42347 (local address 127.0.0.1:53356)
0507 02:09:43.631495 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:43.631505 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0507 02:09:43.631548 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:09:43.631676 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:09:43.631677 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:09:43.631784 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:09:43.631912 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:43.631913 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:43.632676 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:09:43.632677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:09:43.633152 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:43.633154 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:43.633218 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:43.633703 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:09:43.633706 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:09:43.634812 (+ 1106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:09:43.636267 (+ 1455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:09:43.636268 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:09:43.636268 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:09:43.636459 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:09:43.636594 (+ 135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:09:43.636595 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:09:43.636595 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:09:43.636623 (+ 28us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:09:43.636791 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:09:43.636791 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:09:43.636828 (+ 37us) client_negotiation.cc:769] Sending connection context
0507 02:09:43.636908 (+ 80us) client_negotiation.cc:240] Negotiation successful
0507 02:09:43.637002 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20240507 02:09:43.637509 13410 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.131.124" port: 36111 } has no permanent_uuid. Determining permanent_uuid...
I20240507 02:09:43.638139 13414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:43.630522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33448 (local address 127.31.131.124:36111)
0507 02:09:43.630753 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0507 02:09:43.630760 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0507 02:09:43.630777 (+ 17us) server_negotiation.cc:407] Connection header received
0507 02:09:43.630797 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:09:43.630799 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:09:43.630810 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:09:43.630881 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:09:43.631652 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:43.632529 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:09:43.633228 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:43.633413 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:43.635406 (+ 1993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:09:43.635414 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:09:43.635414 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:09:43.635419 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:09:43.637114 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:09:43.637473 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:09:43.637476 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:09:43.637476 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:09:43.637506 (+ 30us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:09:43.637655 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:09:43.637656 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:09:43.637656 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:09:43.637765 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:09:43.637889 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0507 02:09:43.637975 (+ 86us) server_negotiation.cc:299] Negotiation successful
0507 02:09:43.638054 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":115,"threads_started":1}
May 07 02:09:43 dist-test-slave-v0bg krb5kdc[13179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047783, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.124@KRBTEST.COM for kudu/127.31.131.124@KRBTEST.COM
May 07 02:09:43 dist-test-slave-v0bg krb5kdc[13179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:09:43.643394 13412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:43.637694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.124:36111 (local address 127.0.0.1:33450)
0507 02:09:43.637734 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:43.637742 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0507 02:09:43.637775 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:09:43.638044 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:09:43.638044 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:09:43.638153 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:09:43.638292 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:43.638294 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:43.639072 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:09:43.639073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:09:43.639375 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:43.639377 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:43.639435 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:43.639772 (+ 337us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:09:43.639775 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:09:43.640733 (+ 958us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:09:43.642663 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:09:43.642664 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:09:43.642665 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:09:43.642832 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:09:43.642977 (+ 145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:09:43.642977 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:09:43.642978 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:09:43.642999 (+ 21us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:09:43.643173 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:09:43.643174 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:09:43.643213 (+ 39us) client_negotiation.cc:769] Sending connection context
0507 02:09:43.643275 (+ 62us) client_negotiation.cc:240] Negotiation successful
0507 02:09:43.643344 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20240507 02:09:43.643502 13415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:43.637735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33450 (local address 127.31.131.124:36111)
0507 02:09:43.637934 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0507 02:09:43.637935 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:09:43.637947 (+ 12us) server_negotiation.cc:407] Connection header received
0507 02:09:43.637961 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:09:43.637962 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:09:43.637970 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:09:43.638028 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:09:43.638364 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:43.639006 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:09:43.639445 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:43.639591 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:43.640801 (+ 1210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:09:43.640804 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:09:43.640804 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:09:43.640810 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:09:43.642600 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:09:43.642893 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:09:43.642894 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:09:43.642894 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:09:43.642920 (+ 26us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:09:43.643042 (+ 122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:09:43.643042 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:09:43.643043 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:09:43.643123 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:09:43.643170 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0507 02:09:43.643299 (+ 129us) server_negotiation.cc:299] Negotiation successful
0507 02:09:43.643409 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":183,"thread_start_us":68,"threads_started":1}
I20240507 02:09:43.644624 13263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:43.638808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60794 (local address 127.31.131.126:36097)
0507 02:09:43.638850 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0507 02:09:43.638851 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:09:43.638861 (+ 10us) server_negotiation.cc:407] Connection header received
0507 02:09:43.638880 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:09:43.638881 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:09:43.638889 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:09:43.638925 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:09:43.639421 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:43.639960 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:09:43.640438 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:43.640593 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:43.642395 (+ 1802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:09:43.642398 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:09:43.642398 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:09:43.642404 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:09:43.643849 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:09:43.644131 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:09:43.644131 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:09:43.644132 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:09:43.644146 (+ 14us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:09:43.644276 (+ 130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:09:43.644277 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:09:43.644277 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:09:43.644339 (+ 62us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:09:43.644416 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0507 02:09:43.644513 (+ 97us) server_negotiation.cc:299] Negotiation successful
0507 02:09:43.644568 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20240507 02:09:43.644614 13410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b81b4b0655dd46e690e9120e0f2ef20e: Bootstrap starting.
I20240507 02:09:43.645148 13410 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b81b4b0655dd46e690e9120e0f2ef20e: Neither blocks nor log segments found. Creating new log.
I20240507 02:09:43.645411 13410 log.cc:824] T 00000000000000000000000000000000 P b81b4b0655dd46e690e9120e0f2ef20e: Log is configured to *not* fsync() on all Append() calls
I20240507 02:09:43.646104 13410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b81b4b0655dd46e690e9120e0f2ef20e: No bootstrap required, opened a new log
I20240507 02:09:43.647629 13410 raft_consensus.cc:348] T 00000000000000000000000000000000 P b81b4b0655dd46e690e9120e0f2ef20e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4cf873b2b65476eb38261d37c390ad7" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 36097 } } peers { permanent_uuid: "629a4752d61d4b21be97a73698100499" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 42347 } } peers { permanent_uuid: "b81b4b0655dd46e690e9120e0f2ef20e" member_type: VOTER last_known_addr { host: "127.31.131.124" port: 36111 } }
I20240507 02:09:43.647733 13410 raft_consensus.cc:374] T 00000000000000000000000000000000 P b81b4b0655dd46e690e9120e0f2ef20e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:09:43.647754 13410 raft_consensus.cc:729] T 00000000000000000000000000000000 P b81b4b0655dd46e690e9120e0f2ef20e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b81b4b0655dd46e690e9120e0f2ef20e, State: Initialized, Role: FOLLOWER
I20240507 02:09:43.647840 13410 consensus_queue.cc:260] T 00000000000000000000000000000000 P b81b4b0655dd46e690e9120e0f2ef20e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4cf873b2b65476eb38261d37c390ad7" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 36097 } } peers { permanent_uuid: "629a4752d61d4b21be97a73698100499" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 42347 } } peers { permanent_uuid: "b81b4b0655dd46e690e9120e0f2ef20e" member_type: VOTER last_known_addr { host: "127.31.131.124" port: 36111 } }
I20240507 02:09:43.648059 13417 sys_catalog.cc:455] T 00000000000000000000000000000000 P b81b4b0655dd46e690e9120e0f2ef20e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4cf873b2b65476eb38261d37c390ad7" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 36097 } } peers { permanent_uuid: "629a4752d61d4b21be97a73698100499" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 42347 } } peers { permanent_uuid: "b81b4b0655dd46e690e9120e0f2ef20e" member_type: VOTER last_known_addr { host: "127.31.131.124" port: 36111 } } }
I20240507 02:09:43.648097 13410 sys_catalog.cc:564] T 00000000000000000000000000000000 P b81b4b0655dd46e690e9120e0f2ef20e [sys.catalog]: configured and running, proceeding with master startup.
I20240507 02:09:43.648140 13417 sys_catalog.cc:458] T 00000000000000000000000000000000 P b81b4b0655dd46e690e9120e0f2ef20e [sys.catalog]: This master's current role is: FOLLOWER
May 07 02:09:43 dist-test-slave-v0bg krb5kdc[13179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.131.125@KRBTEST.COM
W20240507 02:09:43.650319 13428 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b81b4b0655dd46e690e9120e0f2ef20e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240507 02:09:43.650374 13428 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240507 02:09:43.652072 13337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:43.645448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53366 (local address 127.31.131.125:42347)
0507 02:09:43.645498 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0507 02:09:43.645500 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:09:43.645539 (+ 39us) server_negotiation.cc:407] Connection header received
0507 02:09:43.645602 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:09:43.645603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:09:43.645611 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:09:43.645664 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:09:43.646280 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:43.646702 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:09:43.647322 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:43.647418 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:43.649569 (+ 2151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:09:43.649572 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:09:43.649572 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:09:43.649577 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:09:43.651154 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:09:43.651470 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:09:43.651471 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:09:43.651471 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:09:43.651491 (+ 20us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:09:43.651632 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:09:43.651632 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:09:43.651633 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:09:43.651723 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:09:43.651835 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0507 02:09:43.651919 (+ 84us) server_negotiation.cc:299] Negotiation successful
0507 02:09:43.651990 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20240507 02:09:43.650857 13342 master_runner.cc:186] Error getting master registration for 127.31.131.126:36097: OK, instance_id { permanent_uuid: "c4cf873b2b65476eb38261d37c390ad7" instance_seqno: 1715047783349233 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20240507 02:09:43.653050 13342 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240507 02:09:43.653375 32269 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240507 02:09:43.661101 13415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:43.654728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33462 (local address 127.31.131.124:36111)
0507 02:09:43.654766 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0507 02:09:43.654767 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:09:43.654862 (+ 95us) server_negotiation.cc:407] Connection header received
0507 02:09:43.654942 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:09:43.654942 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:09:43.654951 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:09:43.654974 (+ 23us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:09:43.655670 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:43.656353 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:09:43.656991 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:43.657155 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:43.658439 (+ 1284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:09:43.658442 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:09:43.658442 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:09:43.658448 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:09:43.660059 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:09:43.660379 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:09:43.660380 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:09:43.660380 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:09:43.660403 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:09:43.660550 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:09:43.660550 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:09:43.660550 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:09:43.660656 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:09:43.660724 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0507 02:09:43.660933 (+ 209us) server_negotiation.cc:299] Negotiation successful
0507 02:09:43.661019 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20240507 02:09:43.661211 13263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:43.654483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60810 (local address 127.31.131.126:36097)
0507 02:09:43.654518 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0507 02:09:43.654519 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:09:43.654691 (+ 172us) server_negotiation.cc:407] Connection header received
0507 02:09:43.654780 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:09:43.654781 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:09:43.654790 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:09:43.654816 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:09:43.655740 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:43.656386 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:09:43.657083 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:43.657255 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:43.658677 (+ 1422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:09:43.658681 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:09:43.658681 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:09:43.658687 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:09:43.660202 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:09:43.660551 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:09:43.660552 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:09:43.660553 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:09:43.660576 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:09:43.660725 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:09:43.660726 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:09:43.660726 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:09:43.660817 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:09:43.660876 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0507 02:09:43.661054 (+ 178us) server_negotiation.cc:299] Negotiation successful
0507 02:09:43.661124 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20240507 02:09:43.661845 13337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:43.654575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53374 (local address 127.31.131.125:42347)
0507 02:09:43.654605 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0507 02:09:43.654606 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:09:43.654706 (+ 100us) server_negotiation.cc:407] Connection header received
0507 02:09:43.654793 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:09:43.654794 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:09:43.654802 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:09:43.654829 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:09:43.655670 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:43.656312 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:09:43.656991 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:43.657145 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:43.658909 (+ 1764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:09:43.658913 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:09:43.658913 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:09:43.658919 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:09:43.660843 (+ 1924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:09:43.661164 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:09:43.661166 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:09:43.661166 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:09:43.661187 (+ 21us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:09:43.661352 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:09:43.661353 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:09:43.661353 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:09:43.661437 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:09:43.661491 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0507 02:09:43.661703 (+ 212us) server_negotiation.cc:299] Negotiation successful
0507 02:09:43.661766 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20240507 02:09:43.686087 13260 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.131.124" port: 36111 } attempt: 3
May 07 02:09:43 dist-test-slave-v0bg krb5kdc[13179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047783, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.126@KRBTEST.COM for kudu/127.31.131.124@KRBTEST.COM
I20240507 02:09:43.691567 13415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:43.686433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33464 (local address 127.31.131.124:36111)
0507 02:09:43.686460 (+ 27us) server_negotiation.cc:206] Beginning negotiation
0507 02:09:43.686461 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:09:43.686469 (+ 8us) server_negotiation.cc:407] Connection header received
0507 02:09:43.686512 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:09:43.686512 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:09:43.686520 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:09:43.686588 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:09:43.687002 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:43.687389 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:09:43.688029 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:43.688123 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:43.689628 (+ 1505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:09:43.689631 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:09:43.689631 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:09:43.689636 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:09:43.690809 (+ 1173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:09:43.691098 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:09:43.691099 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:09:43.691099 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:09:43.691113 (+ 14us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:09:43.691240 (+ 127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:09:43.691240 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:09:43.691240 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:09:43.691303 (+ 63us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:09:43.691343 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0507 02:09:43.691463 (+ 120us) server_negotiation.cc:299] Negotiation successful
0507 02:09:43.691511 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20240507 02:09:43.691622 13262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:43.686330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.124:36111 (local address 127.0.0.1:33464)
0507 02:09:43.686426 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:43.686435 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0507 02:09:43.686469 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:09:43.686593 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:09:43.686594 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:09:43.686751 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:09:43.686936 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:43.686937 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:43.687446 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:09:43.687446 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:09:43.687969 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:43.687970 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:43.688036 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:43.688404 (+ 368us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:09:43.688408 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:09:43.689554 (+ 1146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:09:43.690872 (+ 1318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:09:43.690873 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:09:43.690874 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:09:43.691037 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:09:43.691158 (+ 121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:09:43.691159 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:09:43.691159 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:09:43.691185 (+ 26us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:09:43.691347 (+ 162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:09:43.691348 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:09:43.691387 (+ 39us) client_negotiation.cc:769] Sending connection context
0507 02:09:43.691450 (+ 63us) client_negotiation.cc:240] Negotiation successful
0507 02:09:43.691530 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20240507 02:09:43.692714 13260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4cf873b2b65476eb38261d37c390ad7: Bootstrap starting.
I20240507 02:09:43.693210 13260 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4cf873b2b65476eb38261d37c390ad7: Neither blocks nor log segments found. Creating new log.
I20240507 02:09:43.693425 13260 log.cc:824] T 00000000000000000000000000000000 P c4cf873b2b65476eb38261d37c390ad7: Log is configured to *not* fsync() on all Append() calls
I20240507 02:09:43.694039 13260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4cf873b2b65476eb38261d37c390ad7: No bootstrap required, opened a new log
I20240507 02:09:43.695448 13260 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4cf873b2b65476eb38261d37c390ad7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4cf873b2b65476eb38261d37c390ad7" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 36097 } } peers { permanent_uuid: "629a4752d61d4b21be97a73698100499" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 42347 } } peers { permanent_uuid: "b81b4b0655dd46e690e9120e0f2ef20e" member_type: VOTER last_known_addr { host: "127.31.131.124" port: 36111 } }
I20240507 02:09:43.695539 13260 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4cf873b2b65476eb38261d37c390ad7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:09:43.695559 13260 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4cf873b2b65476eb38261d37c390ad7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4cf873b2b65476eb38261d37c390ad7, State: Initialized, Role: FOLLOWER
I20240507 02:09:43.695642 13260 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4cf873b2b65476eb38261d37c390ad7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4cf873b2b65476eb38261d37c390ad7" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 36097 } } peers { permanent_uuid: "629a4752d61d4b21be97a73698100499" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 42347 } } peers { permanent_uuid: "b81b4b0655dd46e690e9120e0f2ef20e" member_type: VOTER last_known_addr { host: "127.31.131.124" port: 36111 } }
I20240507 02:09:43.695899 13260 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4cf873b2b65476eb38261d37c390ad7 [sys.catalog]: configured and running, proceeding with master startup.
I20240507 02:09:43.695868 13438 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4cf873b2b65476eb38261d37c390ad7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4cf873b2b65476eb38261d37c390ad7" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 36097 } } peers { permanent_uuid: "629a4752d61d4b21be97a73698100499" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 42347 } } peers { permanent_uuid: "b81b4b0655dd46e690e9120e0f2ef20e" member_type: VOTER last_known_addr { host: "127.31.131.124" port: 36111 } } }
I20240507 02:09:43.695945 13438 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4cf873b2b65476eb38261d37c390ad7 [sys.catalog]: This master's current role is: FOLLOWER
W20240507 02:09:43.697444 13449 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c4cf873b2b65476eb38261d37c390ad7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240507 02:09:43.697518 13449 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240507 02:09:43.698036 13192 master_runner.cc:186] Error getting master registration for 127.31.131.125:42347: OK, instance_id { permanent_uuid: "629a4752d61d4b21be97a73698100499" instance_seqno: 1715047783476278 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20240507 02:09:43.700214 13192 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240507 02:09:43.703219 13335 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.131.124" port: 36111 } attempt: 3
May 07 02:09:43 dist-test-slave-v0bg krb5kdc[13179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047783, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.125@KRBTEST.COM for kudu/127.31.131.124@KRBTEST.COM
I20240507 02:09:43.709745 13338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:43.703440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.124:36111 (local address 127.0.0.1:33470)
0507 02:09:43.703482 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:43.703492 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0507 02:09:43.703536 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:09:43.703651 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:09:43.703652 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:09:43.703808 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:09:43.703987 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:43.703989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:43.704804 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:09:43.704805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:09:43.705331 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:43.705333 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:43.705404 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:43.705902 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:09:43.705906 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:09:43.707275 (+ 1369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:09:43.708909 (+ 1634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:09:43.708910 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:09:43.708910 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:09:43.709077 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:09:43.709244 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:09:43.709244 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:09:43.709244 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:09:43.709272 (+ 28us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:09:43.709473 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:09:43.709474 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:09:43.709512 (+ 38us) client_negotiation.cc:769] Sending connection context
0507 02:09:43.709574 (+ 62us) client_negotiation.cc:240] Negotiation successful
0507 02:09:43.709654 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20240507 02:09:43.709757 13415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:43.703488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33470 (local address 127.31.131.124:36111)
0507 02:09:43.703544 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0507 02:09:43.703546 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:09:43.703555 (+ 9us) server_negotiation.cc:407] Connection header received
0507 02:09:43.703573 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:09:43.703574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:09:43.703583 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:09:43.703638 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:09:43.704062 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:43.704716 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:09:43.705410 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:43.705583 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:43.707354 (+ 1771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:09:43.707357 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:09:43.707358 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:09:43.707363 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:09:43.708827 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:09:43.709151 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:09:43.709152 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:09:43.709152 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:09:43.709177 (+ 25us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:09:43.709324 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:09:43.709324 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:09:43.709325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:09:43.709420 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:09:43.709477 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0507 02:09:43.709603 (+ 126us) server_negotiation.cc:299] Negotiation successful
0507 02:09:43.709672 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20240507 02:09:43.711499 13335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 629a4752d61d4b21be97a73698100499: Bootstrap starting.
I20240507 02:09:43.712069 13335 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 629a4752d61d4b21be97a73698100499: Neither blocks nor log segments found. Creating new log.
I20240507 02:09:43.712332 13335 log.cc:824] T 00000000000000000000000000000000 P 629a4752d61d4b21be97a73698100499: Log is configured to *not* fsync() on all Append() calls
I20240507 02:09:43.713281 13335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 629a4752d61d4b21be97a73698100499: No bootstrap required, opened a new log
I20240507 02:09:43.715060 13335 raft_consensus.cc:348] T 00000000000000000000000000000000 P 629a4752d61d4b21be97a73698100499 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4cf873b2b65476eb38261d37c390ad7" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 36097 } } peers { permanent_uuid: "629a4752d61d4b21be97a73698100499" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 42347 } } peers { permanent_uuid: "b81b4b0655dd46e690e9120e0f2ef20e" member_type: VOTER last_known_addr { host: "127.31.131.124" port: 36111 } }
I20240507 02:09:43.715227 13335 raft_consensus.cc:374] T 00000000000000000000000000000000 P 629a4752d61d4b21be97a73698100499 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:09:43.715299 13335 raft_consensus.cc:729] T 00000000000000000000000000000000 P 629a4752d61d4b21be97a73698100499 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 629a4752d61d4b21be97a73698100499, State: Initialized, Role: FOLLOWER
I20240507 02:09:43.715443 13335 consensus_queue.cc:260] T 00000000000000000000000000000000 P 629a4752d61d4b21be97a73698100499 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4cf873b2b65476eb38261d37c390ad7" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 36097 } } peers { permanent_uuid: "629a4752d61d4b21be97a73698100499" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 42347 } } peers { permanent_uuid: "b81b4b0655dd46e690e9120e0f2ef20e" member_type: VOTER last_known_addr { host: "127.31.131.124" port: 36111 } }
I20240507 02:09:43.715759 13335 sys_catalog.cc:564] T 00000000000000000000000000000000 P 629a4752d61d4b21be97a73698100499 [sys.catalog]: configured and running, proceeding with master startup.
I20240507 02:09:43.715754 13452 sys_catalog.cc:455] T 00000000000000000000000000000000 P 629a4752d61d4b21be97a73698100499 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4cf873b2b65476eb38261d37c390ad7" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 36097 } } peers { permanent_uuid: "629a4752d61d4b21be97a73698100499" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 42347 } } peers { permanent_uuid: "b81b4b0655dd46e690e9120e0f2ef20e" member_type: VOTER last_known_addr { host: "127.31.131.124" port: 36111 } } }
I20240507 02:09:43.715945 13452 sys_catalog.cc:458] T 00000000000000000000000000000000 P 629a4752d61d4b21be97a73698100499 [sys.catalog]: This master's current role is: FOLLOWER
W20240507 02:09:43.717638 13463 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 629a4752d61d4b21be97a73698100499: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240507 02:09:43.717726 13463 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240507 02:09:43.719175 13267 master_runner.cc:263] No leader master found from master 629a4752d61d4b21be97a73698100499
I20240507 02:09:43.719215 13267 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240507 02:09:43.722879 13438 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4cf873b2b65476eb38261d37c390ad7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 02:09:43.722946 13438 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4cf873b2b65476eb38261d37c390ad7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4cf873b2b65476eb38261d37c390ad7" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 36097 } } peers { permanent_uuid: "629a4752d61d4b21be97a73698100499" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 42347 } } peers { permanent_uuid: "b81b4b0655dd46e690e9120e0f2ef20e" member_type: VOTER last_known_addr { host: "127.31.131.124" port: 36111 } }
I20240507 02:09:43.723129 13438 leader_election.cc:288] T 00000000000000000000000000000000 P c4cf873b2b65476eb38261d37c390ad7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 629a4752d61d4b21be97a73698100499 (127.31.131.125:42347), b81b4b0655dd46e690e9120e0f2ef20e (127.31.131.124:36111)
I20240507 02:09:43.723443 13309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c4cf873b2b65476eb38261d37c390ad7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "629a4752d61d4b21be97a73698100499" is_pre_election: true
I20240507 02:09:43.723548 13309 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 629a4752d61d4b21be97a73698100499 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4cf873b2b65476eb38261d37c390ad7 in term 0.
I20240507 02:09:43.723766 13209 leader_election.cc:302] T 00000000000000000000000000000000 P c4cf873b2b65476eb38261d37c390ad7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 629a4752d61d4b21be97a73698100499, c4cf873b2b65476eb38261d37c390ad7; no voters:
I20240507 02:09:43.723932 13438 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4cf873b2b65476eb38261d37c390ad7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 02:09:43.723985 13438 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4cf873b2b65476eb38261d37c390ad7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 02:09:43.724007 13438 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4cf873b2b65476eb38261d37c390ad7 [term 0 FOLLOWER]: Advancing to term 1
W20240507 02:09:43.724174 13288 tablet.cc:2369] T 00000000000000000000000000000000 P 629a4752d61d4b21be97a73698100499: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240507 02:09:43.724825 13438 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4cf873b2b65476eb38261d37c390ad7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4cf873b2b65476eb38261d37c390ad7" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 36097 } } peers { permanent_uuid: "629a4752d61d4b21be97a73698100499" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 42347 } } peers { permanent_uuid: "b81b4b0655dd46e690e9120e0f2ef20e" member_type: VOTER last_known_addr { host: "127.31.131.124" port: 36111 } }
I20240507 02:09:43.724925 13438 leader_election.cc:288] T 00000000000000000000000000000000 P c4cf873b2b65476eb38261d37c390ad7 [CANDIDATE]: Term 1 election: Requested vote from peers 629a4752d61d4b21be97a73698100499 (127.31.131.125:42347), b81b4b0655dd46e690e9120e0f2ef20e (127.31.131.124:36111)
I20240507 02:09:43.725299 13309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c4cf873b2b65476eb38261d37c390ad7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "629a4752d61d4b21be97a73698100499"
I20240507 02:09:43.725356 13309 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 629a4752d61d4b21be97a73698100499 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:09:43.726128 13309 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 629a4752d61d4b21be97a73698100499 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4cf873b2b65476eb38261d37c390ad7 in term 1.
I20240507 02:09:43.726326 13209 leader_election.cc:302] T 00000000000000000000000000000000 P c4cf873b2b65476eb38261d37c390ad7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 629a4752d61d4b21be97a73698100499, c4cf873b2b65476eb38261d37c390ad7; no voters:
I20240507 02:09:43.726660 13383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c4cf873b2b65476eb38261d37c390ad7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b81b4b0655dd46e690e9120e0f2ef20e"
I20240507 02:09:43.726752 13383 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b81b4b0655dd46e690e9120e0f2ef20e [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:09:43.727481 13383 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b81b4b0655dd46e690e9120e0f2ef20e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4cf873b2b65476eb38261d37c390ad7 in term 1.
I20240507 02:09:43.727627 13384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c4cf873b2b65476eb38261d37c390ad7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b81b4b0655dd46e690e9120e0f2ef20e" is_pre_election: true
I20240507 02:09:43.727681 13384 raft_consensus.cc:2362] T 00000000000000000000000000000000 P b81b4b0655dd46e690e9120e0f2ef20e [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate c4cf873b2b65476eb38261d37c390ad7 in term 1. Re-sending same reply.
I20240507 02:09:43.728622 13438 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4cf873b2b65476eb38261d37c390ad7 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:09:43.728747 13438 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4cf873b2b65476eb38261d37c390ad7 [term 1 LEADER]: Becoming Leader. State: Replica: c4cf873b2b65476eb38261d37c390ad7, State: Running, Role: LEADER
I20240507 02:09:43.728812 13438 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4cf873b2b65476eb38261d37c390ad7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4cf873b2b65476eb38261d37c390ad7" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 36097 } } peers { permanent_uuid: "629a4752d61d4b21be97a73698100499" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 42347 } } peers { permanent_uuid: "b81b4b0655dd46e690e9120e0f2ef20e" member_type: VOTER last_known_addr { host: "127.31.131.124" port: 36111 } }
I20240507 02:09:43.729084 13438 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4cf873b2b65476eb38261d37c390ad7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4cf873b2b65476eb38261d37c390ad7. Latest consensus state: current_term: 1 leader_uuid: "c4cf873b2b65476eb38261d37c390ad7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4cf873b2b65476eb38261d37c390ad7" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 36097 } } peers { permanent_uuid: "629a4752d61d4b21be97a73698100499" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 42347 } } peers { permanent_uuid: "b81b4b0655dd46e690e9120e0f2ef20e" member_type: VOTER last_known_addr { host: "127.31.131.124" port: 36111 } } }
I20240507 02:09:43.729143 13438 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4cf873b2b65476eb38261d37c390ad7 [sys.catalog]: This master's current role is: LEADER
I20240507 02:09:43.729451 13467 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 02:09:43.729640 13467 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240507 02:09:43.731473 13384 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b81b4b0655dd46e690e9120e0f2ef20e [term 1 FOLLOWER]: Refusing update from remote peer c4cf873b2b65476eb38261d37c390ad7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 02:09:43.731662 13309 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 629a4752d61d4b21be97a73698100499 [term 1 FOLLOWER]: Refusing update from remote peer c4cf873b2b65476eb38261d37c390ad7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 02:09:43.732025 13438 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c4cf873b2b65476eb38261d37c390ad7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "629a4752d61d4b21be97a73698100499" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 42347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 02:09:43.732136 13438 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c4cf873b2b65476eb38261d37c390ad7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b81b4b0655dd46e690e9120e0f2ef20e" member_type: VOTER last_known_addr { host: "127.31.131.124" port: 36111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 02:09:43.733124 13452 sys_catalog.cc:455] T 00000000000000000000000000000000 P 629a4752d61d4b21be97a73698100499 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4cf873b2b65476eb38261d37c390ad7. Latest consensus state: current_term: 1 leader_uuid: "c4cf873b2b65476eb38261d37c390ad7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4cf873b2b65476eb38261d37c390ad7" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 36097 } } peers { permanent_uuid: "629a4752d61d4b21be97a73698100499" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 42347 } } peers { permanent_uuid: "b81b4b0655dd46e690e9120e0f2ef20e" member_type: VOTER last_known_addr { host: "127.31.131.124" port: 36111 } } }
I20240507 02:09:43.733193 13452 sys_catalog.cc:458] T 00000000000000000000000000000000 P 629a4752d61d4b21be97a73698100499 [sys.catalog]: This master's current role is: FOLLOWER
I20240507 02:09:43.734051 13417 sys_catalog.cc:455] T 00000000000000000000000000000000 P b81b4b0655dd46e690e9120e0f2ef20e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4cf873b2b65476eb38261d37c390ad7. Latest consensus state: current_term: 1 leader_uuid: "c4cf873b2b65476eb38261d37c390ad7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4cf873b2b65476eb38261d37c390ad7" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 36097 } } peers { permanent_uuid: "629a4752d61d4b21be97a73698100499" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 42347 } } peers { permanent_uuid: "b81b4b0655dd46e690e9120e0f2ef20e" member_type: VOTER last_known_addr { host: "127.31.131.124" port: 36111 } } }
I20240507 02:09:43.734122 13417 sys_catalog.cc:458] T 00000000000000000000000000000000 P b81b4b0655dd46e690e9120e0f2ef20e [sys.catalog]: This master's current role is: FOLLOWER
I20240507 02:09:43.735723 13438 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4cf873b2b65476eb38261d37c390ad7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c4cf873b2b65476eb38261d37c390ad7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4cf873b2b65476eb38261d37c390ad7" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 36097 } } peers { permanent_uuid: "629a4752d61d4b21be97a73698100499" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 42347 } } peers { permanent_uuid: "b81b4b0655dd46e690e9120e0f2ef20e" member_type: VOTER last_known_addr { host: "127.31.131.124" port: 36111 } } }
I20240507 02:09:43.735791 13438 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4cf873b2b65476eb38261d37c390ad7 [sys.catalog]: This master's current role is: LEADER
I20240507 02:09:43.736198 13452 sys_catalog.cc:455] T 00000000000000000000000000000000 P 629a4752d61d4b21be97a73698100499 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c4cf873b2b65476eb38261d37c390ad7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4cf873b2b65476eb38261d37c390ad7" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 36097 } } peers { permanent_uuid: "629a4752d61d4b21be97a73698100499" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 42347 } } peers { permanent_uuid: "b81b4b0655dd46e690e9120e0f2ef20e" member_type: VOTER last_known_addr { host: "127.31.131.124" port: 36111 } } }
I20240507 02:09:43.736274 13452 sys_catalog.cc:458] T 00000000000000000000000000000000 P 629a4752d61d4b21be97a73698100499 [sys.catalog]: This master's current role is: FOLLOWER
I20240507 02:09:43.736496 13468 mvcc.cc:205] Tried to move back new op lower bound from 7024835722161770496 to 7024835722153603072. Current Snapshot: MvccSnapshot[applied={T|T < 7024835722161770496}]
I20240507 02:09:43.736625 13469 mvcc.cc:205] Tried to move back new op lower bound from 7024835722161770496 to 7024835722153603072. Current Snapshot: MvccSnapshot[applied={T|T < 7024835722161770496}]
I20240507 02:09:43.737341 13438 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4cf873b2b65476eb38261d37c390ad7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c4cf873b2b65476eb38261d37c390ad7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4cf873b2b65476eb38261d37c390ad7" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 36097 } } peers { permanent_uuid: "629a4752d61d4b21be97a73698100499" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 42347 } } peers { permanent_uuid: "b81b4b0655dd46e690e9120e0f2ef20e" member_type: VOTER last_known_addr { host: "127.31.131.124" port: 36111 } } }
I20240507 02:09:43.737407 13438 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4cf873b2b65476eb38261d37c390ad7 [sys.catalog]: This master's current role is: LEADER
I20240507 02:09:43.737732 13417 sys_catalog.cc:455] T 00000000000000000000000000000000 P b81b4b0655dd46e690e9120e0f2ef20e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c4cf873b2b65476eb38261d37c390ad7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4cf873b2b65476eb38261d37c390ad7" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 36097 } } peers { permanent_uuid: "629a4752d61d4b21be97a73698100499" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 42347 } } peers { permanent_uuid: "b81b4b0655dd46e690e9120e0f2ef20e" member_type: VOTER last_known_addr { host: "127.31.131.124" port: 36111 } } }
I20240507 02:09:43.737799 13417 sys_catalog.cc:458] T 00000000000000000000000000000000 P b81b4b0655dd46e690e9120e0f2ef20e [sys.catalog]: This master's current role is: FOLLOWER
I20240507 02:09:43.738315 13471 mvcc.cc:205] Tried to move back new op lower bound from 7024835722161770496 to 7024835722153603072. Current Snapshot: MvccSnapshot[applied={T|T < 7024835722161770496}]
I20240507 02:09:43.738865 13467 catalog_manager.cc:1347] Generated new cluster ID: 1017f98bf8104025bd592554237b4126
I20240507 02:09:43.738919 13467 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240507 02:09:43.757117 13467 catalog_manager.cc:1370] Generated new certificate authority record
I20240507 02:09:43.757797 13467 catalog_manager.cc:1504] Loading token signing keys...
I20240507 02:09:43.767534 13467 catalog_manager.cc:5857] T 00000000000000000000000000000000 P c4cf873b2b65476eb38261d37c390ad7: Generated new TSK 0
I20240507 02:09:43.767690 13467 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240507 02:09:44.650725 13428 catalog_manager.cc:1259] Loaded cluster ID: 1017f98bf8104025bd592554237b4126
I20240507 02:09:44.650794 13428 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b81b4b0655dd46e690e9120e0f2ef20e: loading cluster ID for follower catalog manager: success
I20240507 02:09:44.651610 13428 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b81b4b0655dd46e690e9120e0f2ef20e: acquiring CA information for follower catalog manager: success
I20240507 02:09:44.651861 13428 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b81b4b0655dd46e690e9120e0f2ef20e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240507 02:09:44.718123 13463 catalog_manager.cc:1259] Loaded cluster ID: 1017f98bf8104025bd592554237b4126
I20240507 02:09:44.718184 13463 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 629a4752d61d4b21be97a73698100499: loading cluster ID for follower catalog manager: success
I20240507 02:09:44.719069 13463 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 629a4752d61d4b21be97a73698100499: acquiring CA information for follower catalog manager: success
I20240507 02:09:44.719329 13463 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 629a4752d61d4b21be97a73698100499: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240507 02:09:44.828817 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 13192
I20240507 02:09:44.833492 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 13267
I20240507 02:09:44.837451 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 13342
2024-05-07T02:09:44Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2581 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1715047604443657-32269-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 07 02:09:44 dist-test-slave-v0bg krb5kdc[13485](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 07 02:09:44 dist-test-slave-v0bg krb5kdc[13485](info): set up 2 sockets
May 07 02:09:44 dist-test-slave-v0bg krb5kdc[13485](info): commencing operation
krb5kdc: starting...
W20240507 02:09:45.723100 32269 mini_kdc.cc:121] Time spent starting KDC: real 0.868s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1715047604443657-32269-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1715047604443657-32269-0/krb5kdc/test-user.keytab.
May 07 02:09:45 dist-test-slave-v0bg krb5kdc[13485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-07T02:09:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-07T02:09:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.131.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab.
Entry for principal kudu/127.31.131.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab.
I20240507 02:09:45.821055 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1715047604443657-32269-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1715047604443657-32269-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1715047604443657-32269-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:36605
--webserver_interface=127.31.131.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab
--principal=kudu/127.31.131.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:44621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.131.126:36605
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:09:45.893087 13498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:09:45.893242 13498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:09:45.893260 13498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:09:45.894488 13498 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240507 02:09:45.894526 13498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:09:45.894538 13498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:09:45.894567 13498 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240507 02:09:45.894591 13498 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240507 02:09:45.896055 13498 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:44621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1715047604443657-32269-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1715047604443657-32269-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.131.126:36605
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab
--principal=kudu/127.31.131.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:36605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.131.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13498
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1715047604443657-32269-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:09:45.896203 13498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:09:45.896334 13498 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:09:45.899719 13504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:09:45.899801 13498 server_base.cc:1034] running on GCE node
W20240507 02:09:45.899762 13503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:09:45.899878 13506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:09:45.900103 13498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:09:45.900305 13498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:09:45.901434 13498 hybrid_clock.cc:648] HybridClock initialized: now 1715047785901418 us; error 33 us; skew 500 ppm
May 07 02:09:45 dist-test-slave-v0bg krb5kdc[13485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047785, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:09:45.903484 13498 init.cc:377] Logged in from keytab as kudu/127.31.131.126@KRBTEST.COM (short username kudu)
I20240507 02:09:45.903800 13498 webserver.cc:460] Webserver started at http://127.31.131.126:32843/ using document root <none> and password file <none>
I20240507 02:09:45.903928 13498 fs_manager.cc:359] Metadata directory not provided
I20240507 02:09:45.903960 13498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:09:45.904031 13498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:09:45.904832 13498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1715047604443657-32269-0/minicluster-data/master-0/data/instance:
uuid: "02d9bd40820e499f97239129e64afcaa"
format_stamp: "Formatted at 2024-05-07 02:09:45 on dist-test-slave-v0bg"
server_key: "cd7e7eddd81404f4dbc4c364a8efc235"
server_key_iv: "7371649905b98d2a71049e940209c2d6"
server_key_version: "encryptionkey@0"
I20240507 02:09:45.905064 13498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1715047604443657-32269-0/minicluster-data/master-0/wal/instance:
uuid: "02d9bd40820e499f97239129e64afcaa"
format_stamp: "Formatted at 2024-05-07 02:09:45 on dist-test-slave-v0bg"
server_key: "cd7e7eddd81404f4dbc4c364a8efc235"
server_key_iv: "7371649905b98d2a71049e940209c2d6"
server_key_version: "encryptionkey@0"
I20240507 02:09:45.905980 13498 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240507 02:09:45.906491 13513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:09:45.906638 13498 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240507 02:09:45.906704 13498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1715047604443657-32269-0/minicluster-data/master-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1715047604443657-32269-0/minicluster-data/master-0/wal
uuid: "02d9bd40820e499f97239129e64afcaa"
format_stamp: "Formatted at 2024-05-07 02:09:45 on dist-test-slave-v0bg"
server_key: "cd7e7eddd81404f4dbc4c364a8efc235"
server_key_iv: "7371649905b98d2a71049e940209c2d6"
server_key_version: "encryptionkey@0"
I20240507 02:09:45.906767 13498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1715047604443657-32269-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1715047604443657-32269-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1715047604443657-32269-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:09:45.917821 13498 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:09:45.919350 13498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:09:45.919459 13498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:09:45.922856 13498 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.126:36605
I20240507 02:09:45.922884 13565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.126:36605 every 8 connection(s)
I20240507 02:09:45.923138 13498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
I20240507 02:09:45.923727 13566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:09:45.924031 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 13498
I20240507 02:09:45.924111 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1715047604443657-32269-0/minicluster-data/master-0/wal/instance
I20240507 02:09:45.924299 32269 external_mini_cluster.cc:1428] Setting key e75454f7f23e2edef1eee94e82c5e81f
I20240507 02:09:45.926076 13566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02d9bd40820e499f97239129e64afcaa: Bootstrap starting.
I20240507 02:09:45.926569 13566 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02d9bd40820e499f97239129e64afcaa: Neither blocks nor log segments found. Creating new log.
I20240507 02:09:45.926810 13566 log.cc:824] T 00000000000000000000000000000000 P 02d9bd40820e499f97239129e64afcaa: Log is configured to *not* fsync() on all Append() calls
I20240507 02:09:45.927336 13566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02d9bd40820e499f97239129e64afcaa: No bootstrap required, opened a new log
I20240507 02:09:45.928885 13566 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02d9bd40820e499f97239129e64afcaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02d9bd40820e499f97239129e64afcaa" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 36605 } }
I20240507 02:09:45.928987 13566 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02d9bd40820e499f97239129e64afcaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:09:45.929010 13566 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02d9bd40820e499f97239129e64afcaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02d9bd40820e499f97239129e64afcaa, State: Initialized, Role: FOLLOWER
May 07 02:09:45 dist-test-slave-v0bg krb5kdc[13485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:09:45.929095 13566 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02d9bd40820e499f97239129e64afcaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02d9bd40820e499f97239129e64afcaa" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 36605 } }
I20240507 02:09:45.929163 13566 raft_consensus.cc:388] T 00000000000000000000000000000000 P 02d9bd40820e499f97239129e64afcaa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 02:09:45.929201 13566 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02d9bd40820e499f97239129e64afcaa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 02:09:45.929234 13566 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02d9bd40820e499f97239129e64afcaa [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:09:45.929776 13566 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02d9bd40820e499f97239129e64afcaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02d9bd40820e499f97239129e64afcaa" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 36605 } }
I20240507 02:09:45.929872 13566 leader_election.cc:302] T 00000000000000000000000000000000 P 02d9bd40820e499f97239129e64afcaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02d9bd40820e499f97239129e64afcaa; no voters:
I20240507 02:09:45.930016 13566 leader_election.cc:288] T 00000000000000000000000000000000 P 02d9bd40820e499f97239129e64afcaa [CANDIDATE]: Term 1 election: Requested vote from peers
I20240507 02:09:45.930135 13571 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02d9bd40820e499f97239129e64afcaa [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:09:45.930235 13566 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02d9bd40820e499f97239129e64afcaa [sys.catalog]: configured and running, proceeding with master startup.
I20240507 02:09:45.930313 13571 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02d9bd40820e499f97239129e64afcaa [term 1 LEADER]: Becoming Leader. State: Replica: 02d9bd40820e499f97239129e64afcaa, State: Running, Role: LEADER
I20240507 02:09:45.930421 13571 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02d9bd40820e499f97239129e64afcaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02d9bd40820e499f97239129e64afcaa" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 36605 } }
I20240507 02:09:45.930796 13572 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02d9bd40820e499f97239129e64afcaa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02d9bd40820e499f97239129e64afcaa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02d9bd40820e499f97239129e64afcaa" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 36605 } } }
I20240507 02:09:45.930876 13572 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02d9bd40820e499f97239129e64afcaa [sys.catalog]: This master's current role is: LEADER
I20240507 02:09:45.931041 13573 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02d9bd40820e499f97239129e64afcaa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02d9bd40820e499f97239129e64afcaa. Latest consensus state: current_term: 1 leader_uuid: "02d9bd40820e499f97239129e64afcaa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02d9bd40820e499f97239129e64afcaa" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 36605 } } }
I20240507 02:09:45.931070 13580 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 02:09:45.931100 13573 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02d9bd40820e499f97239129e64afcaa [sys.catalog]: This master's current role is: LEADER
I20240507 02:09:45.931502 13580 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240507 02:09:45.932679 13588 mvcc.cc:205] Tried to move back new op lower bound from 7024835731178344448 to 7024835731171315712. Current Snapshot: MvccSnapshot[applied={T|T < 7024835731178344448}]
I20240507 02:09:45.933032 13580 catalog_manager.cc:1347] Generated new cluster ID: d956b1c5a0c44bdb80cfe2232f7dd413
I20240507 02:09:45.933092 13580 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240507 02:09:45.934706 13569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:45.924712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52604 (local address 127.31.131.126:36605)
0507 02:09:45.924839 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0507 02:09:45.924847 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0507 02:09:45.924862 (+ 15us) server_negotiation.cc:407] Connection header received
0507 02:09:45.925206 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:09:45.925210 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:09:45.925245 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:09:45.925347 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:09:45.926197 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:45.926979 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:09:45.927529 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:45.927749 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:45.929628 (+ 1879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:09:45.929635 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:09:45.929636 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:09:45.929648 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:09:45.931652 (+ 2004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:09:45.932039 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:09:45.932042 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:09:45.932042 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:09:45.932100 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:09:45.932379 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:09:45.932380 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:09:45.932380 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:09:45.932600 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:09:45.932783 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0507 02:09:45.932865 (+ 82us) server_negotiation.cc:299] Negotiation successful
0507 02:09:45.932974 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":58,"threads_started":1}
I20240507 02:09:45.935566 32269 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240507 02:09:45.945876 13580 catalog_manager.cc:1370] Generated new certificate authority record
I20240507 02:09:45.946331 13580 catalog_manager.cc:1504] Loading token signing keys...
I20240507 02:09:45.949934 13580 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 02d9bd40820e499f97239129e64afcaa: Generated new TSK 0
I20240507 02:09:45.950063 13580 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240507 02:09:45.988963 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 13498
2024-05-07T02:09:45Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1150 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 07 02:09:46 dist-test-slave-v0bg krb5kdc[13593](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 07 02:09:46 dist-test-slave-v0bg krb5kdc[13593](info): set up 2 sockets
May 07 02:09:46 dist-test-slave-v0bg krb5kdc[13593](info): commencing operation
krb5kdc: starting...
W20240507 02:09:46.871263 32269 mini_kdc.cc:121] Time spent starting KDC: real 0.866s user 0.002s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/krb5kdc/test-user.keytab.
May 07 02:09:46 dist-test-slave-v0bg krb5kdc[13593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047786, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-07T02:09:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-07T02:09:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.131.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab.
Entry for principal kudu/127.31.131.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab.
I20240507 02:09:46.968959 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:36547
--webserver_interface=127.31.131.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab
--principal=kudu/127.31.131.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:44929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.131.126:36547
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:09:47.042011 13606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:09:47.042184 13606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:09:47.042212 13606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:09:47.043442 13606 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240507 02:09:47.043479 13606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:09:47.043498 13606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:09:47.043512 13606 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240507 02:09:47.043526 13606 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240507 02:09:47.044910 13606 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:44929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.131.126:36547
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab
--principal=kudu/127.31.131.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:36547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.131.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13606
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:09:47.045099 13606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:09:47.045258 13606 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:09:47.048728 13611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:09:47.048750 13614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:09:47.048918 13612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:09:47.048933 13606 server_base.cc:1034] running on GCE node
I20240507 02:09:47.049233 13606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:09:47.049432 13606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:09:47.050590 13606 hybrid_clock.cc:648] HybridClock initialized: now 1715047787050575 us; error 25 us; skew 500 ppm
May 07 02:09:47 dist-test-slave-v0bg krb5kdc[13593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047787, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:09:47.052732 13606 init.cc:377] Logged in from keytab as kudu/127.31.131.126@KRBTEST.COM (short username kudu)
I20240507 02:09:47.053087 13606 webserver.cc:460] Webserver started at http://127.31.131.126:40679/ using document root <none> and password file <none>
I20240507 02:09:47.053232 13606 fs_manager.cc:359] Metadata directory not provided
I20240507 02:09:47.053270 13606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:09:47.053365 13606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:09:47.054255 13606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/master-0/data/instance:
uuid: "b3f5a6f84b18408c94e6a9c6e49ce384"
format_stamp: "Formatted at 2024-05-07 02:09:47 on dist-test-slave-v0bg"
server_key: "784f36bbe8cea968188f2cc04d2c7033"
server_key_iv: "8af29a7f465f9744faff075c642b1967"
server_key_version: "encryptionkey@0"
I20240507 02:09:47.054523 13606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/master-0/wal/instance:
uuid: "b3f5a6f84b18408c94e6a9c6e49ce384"
format_stamp: "Formatted at 2024-05-07 02:09:47 on dist-test-slave-v0bg"
server_key: "784f36bbe8cea968188f2cc04d2c7033"
server_key_iv: "8af29a7f465f9744faff075c642b1967"
server_key_version: "encryptionkey@0"
I20240507 02:09:47.055523 13606 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240507 02:09:47.056087 13621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:09:47.056221 13606 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240507 02:09:47.056288 13606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/master-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/master-0/wal
uuid: "b3f5a6f84b18408c94e6a9c6e49ce384"
format_stamp: "Formatted at 2024-05-07 02:09:47 on dist-test-slave-v0bg"
server_key: "784f36bbe8cea968188f2cc04d2c7033"
server_key_iv: "8af29a7f465f9744faff075c642b1967"
server_key_version: "encryptionkey@0"
I20240507 02:09:47.056339 13606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:09:47.070029 13606 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:09:47.071799 13606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:09:47.071897 13606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:09:47.075008 13606 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.126:36547
I20240507 02:09:47.075035 13673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.126:36547 every 8 connection(s)
I20240507 02:09:47.075281 13606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
I20240507 02:09:47.075923 13674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:09:47.077657 13674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3f5a6f84b18408c94e6a9c6e49ce384: Bootstrap starting.
I20240507 02:09:47.078114 13674 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3f5a6f84b18408c94e6a9c6e49ce384: Neither blocks nor log segments found. Creating new log.
I20240507 02:09:47.078311 13674 log.cc:824] T 00000000000000000000000000000000 P b3f5a6f84b18408c94e6a9c6e49ce384: Log is configured to *not* fsync() on all Append() calls
I20240507 02:09:47.078882 13674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3f5a6f84b18408c94e6a9c6e49ce384: No bootstrap required, opened a new log
I20240507 02:09:47.080200 13674 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3f5a6f84b18408c94e6a9c6e49ce384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3f5a6f84b18408c94e6a9c6e49ce384" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 36547 } }
I20240507 02:09:47.080302 13674 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3f5a6f84b18408c94e6a9c6e49ce384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:09:47.080318 13674 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3f5a6f84b18408c94e6a9c6e49ce384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3f5a6f84b18408c94e6a9c6e49ce384, State: Initialized, Role: FOLLOWER
I20240507 02:09:47.080395 13674 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3f5a6f84b18408c94e6a9c6e49ce384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3f5a6f84b18408c94e6a9c6e49ce384" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 36547 } }
I20240507 02:09:47.080441 13674 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3f5a6f84b18408c94e6a9c6e49ce384 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 02:09:47.080458 13674 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3f5a6f84b18408c94e6a9c6e49ce384 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 02:09:47.080479 13674 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3f5a6f84b18408c94e6a9c6e49ce384 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:09:47.080943 13674 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3f5a6f84b18408c94e6a9c6e49ce384 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3f5a6f84b18408c94e6a9c6e49ce384" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 36547 } }
I20240507 02:09:47.081019 13674 leader_election.cc:302] T 00000000000000000000000000000000 P b3f5a6f84b18408c94e6a9c6e49ce384 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3f5a6f84b18408c94e6a9c6e49ce384; no voters:
I20240507 02:09:47.081146 13674 leader_election.cc:288] T 00000000000000000000000000000000 P b3f5a6f84b18408c94e6a9c6e49ce384 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240507 02:09:47.081305 13677 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3f5a6f84b18408c94e6a9c6e49ce384 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:09:47.081347 13674 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3f5a6f84b18408c94e6a9c6e49ce384 [sys.catalog]: configured and running, proceeding with master startup.
I20240507 02:09:47.081459 13677 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3f5a6f84b18408c94e6a9c6e49ce384 [term 1 LEADER]: Becoming Leader. State: Replica: b3f5a6f84b18408c94e6a9c6e49ce384, State: Running, Role: LEADER
I20240507 02:09:47.081539 13677 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3f5a6f84b18408c94e6a9c6e49ce384 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3f5a6f84b18408c94e6a9c6e49ce384" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 36547 } }
I20240507 02:09:47.081861 13678 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3f5a6f84b18408c94e6a9c6e49ce384 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3f5a6f84b18408c94e6a9c6e49ce384" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3f5a6f84b18408c94e6a9c6e49ce384" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 36547 } } }
I20240507 02:09:47.081950 13678 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3f5a6f84b18408c94e6a9c6e49ce384 [sys.catalog]: This master's current role is: LEADER
I20240507 02:09:47.082022 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 13606
I20240507 02:09:47.082116 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/master-0/wal/instance
I20240507 02:09:47.082324 32269 external_mini_cluster.cc:1428] Setting key 52651c91c2e4834232a506ea67065a19
I20240507 02:09:47.082165 13679 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3f5a6f84b18408c94e6a9c6e49ce384 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3f5a6f84b18408c94e6a9c6e49ce384. Latest consensus state: current_term: 1 leader_uuid: "b3f5a6f84b18408c94e6a9c6e49ce384" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3f5a6f84b18408c94e6a9c6e49ce384" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 36547 } } }
I20240507 02:09:47.082403 13679 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3f5a6f84b18408c94e6a9c6e49ce384 [sys.catalog]: This master's current role is: LEADER
I20240507 02:09:47.082512 13686 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 02:09:47.082967 13686 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240507 02:09:47.084249 13686 catalog_manager.cc:1347] Generated new cluster ID: a212f29e4e124c83a118b6754b8cae85
I20240507 02:09:47.084298 13686 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
May 07 02:09:47 dist-test-slave-v0bg krb5kdc[13593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047786, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:09:47.095588 13696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:47.082848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41842 (local address 127.31.131.126:36547)
0507 02:09:47.086647 (+ 3799us) server_negotiation.cc:206] Beginning negotiation
0507 02:09:47.086658 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0507 02:09:47.086678 (+ 20us) server_negotiation.cc:407] Connection header received
0507 02:09:47.086768 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:09:47.086773 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:09:47.086817 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:09:47.086948 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:09:47.087863 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:47.089050 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:09:47.089782 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:47.090023 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:47.092300 (+ 2277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:09:47.092305 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:09:47.092306 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:09:47.092323 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:09:47.094184 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:09:47.094534 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:09:47.094536 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:09:47.094537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:09:47.094617 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:09:47.094767 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:09:47.094768 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:09:47.094768 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:09:47.094970 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:09:47.095083 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0507 02:09:47.095209 (+ 126us) server_negotiation.cc:299] Negotiation successful
0507 02:09:47.095411 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3760,"thread_start_us":66,"threads_started":1}
I20240507 02:09:47.098522 13686 catalog_manager.cc:1370] Generated new certificate authority record
I20240507 02:09:47.098986 13686 catalog_manager.cc:1504] Loading token signing keys...
I20240507 02:09:47.109078 13686 catalog_manager.cc:5857] T 00000000000000000000000000000000 P b3f5a6f84b18408c94e6a9c6e49ce384: Generated new TSK 0
I20240507 02:09:47.109220 13686 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.131.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab.
Entry for principal kudu/127.31.131.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab.
I20240507 02:09:47.134812 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.65:0
--local_ip_for_outbound_sockets=127.31.131.65
--webserver_interface=127.31.131.65
--webserver_port=0
--tserver_master_addrs=127.31.131.126:36547
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab
--principal=kudu/127.31.131.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:44929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:09:47.207288 13700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:09:47.207444 13700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:09:47.207473 13700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:09:47.208664 13700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:09:47.208711 13700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:09:47.208743 13700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.65
I20240507 02:09:47.210121 13700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:44929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab
--principal=kudu/127.31.131.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.131.65
--webserver_port=0
--tserver_master_addrs=127.31.131.126:36547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.65
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:09:47.210299 13700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:09:47.210450 13700 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:09:47.213989 13705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:09:47.213984 13706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:09:47.214164 13700 server_base.cc:1034] running on GCE node
W20240507 02:09:47.214205 13708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:09:47.214417 13700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:09:47.214644 13700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:09:47.215801 13700 hybrid_clock.cc:648] HybridClock initialized: now 1715047787215782 us; error 31 us; skew 500 ppm
May 07 02:09:47 dist-test-slave-v0bg krb5kdc[13593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047787, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:09:47.218390 13700 init.cc:377] Logged in from keytab as kudu/127.31.131.65@KRBTEST.COM (short username kudu)
I20240507 02:09:47.218823 13700 webserver.cc:460] Webserver started at http://127.31.131.65:41741/ using document root <none> and password file <none>
I20240507 02:09:47.218999 13700 fs_manager.cc:359] Metadata directory not provided
I20240507 02:09:47.219043 13700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:09:47.219136 13700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:09:47.220240 13700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-0/data/instance:
uuid: "1bec15577655406baa80dd159df72708"
format_stamp: "Formatted at 2024-05-07 02:09:47 on dist-test-slave-v0bg"
server_key: "8632f4ad1e8414ee6fb17ee667ac2f6e"
server_key_iv: "ec4f18d4430225476621a09832fcd7ea"
server_key_version: "encryptionkey@0"
I20240507 02:09:47.220546 13700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-0/wal/instance:
uuid: "1bec15577655406baa80dd159df72708"
format_stamp: "Formatted at 2024-05-07 02:09:47 on dist-test-slave-v0bg"
server_key: "8632f4ad1e8414ee6fb17ee667ac2f6e"
server_key_iv: "ec4f18d4430225476621a09832fcd7ea"
server_key_version: "encryptionkey@0"
I20240507 02:09:47.221793 13700 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240507 02:09:47.222517 13715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:09:47.222694 13700 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240507 02:09:47.222765 13700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-0/wal
uuid: "1bec15577655406baa80dd159df72708"
format_stamp: "Formatted at 2024-05-07 02:09:47 on dist-test-slave-v0bg"
server_key: "8632f4ad1e8414ee6fb17ee667ac2f6e"
server_key_iv: "ec4f18d4430225476621a09832fcd7ea"
server_key_version: "encryptionkey@0"
I20240507 02:09:47.222826 13700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:09:47.229391 13700 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:09:47.230736 13700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:09:47.230823 13700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:09:47.230979 13700 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:09:47.231230 13700 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:09:47.231261 13700 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:09:47.231284 13700 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:09:47.231300 13700 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:09:47.236099 13700 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.65:33661
I20240507 02:09:47.236150 13828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.65:33661 every 8 connection(s)
I20240507 02:09:47.236371 13700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
I20240507 02:09:47.238116 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 13700
I20240507 02:09:47.238242 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-0/wal/instance
I20240507 02:09:47.238425 32269 external_mini_cluster.cc:1428] Setting key ac18de8734ae3ec4459b54cc4d860544
May 07 02:09:47 dist-test-slave-v0bg krb5kdc[13593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047787, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.65@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:09:47.244143 13696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:47.236943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:58295 (local address 127.31.131.126:36547)
0507 02:09:47.237001 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0507 02:09:47.237003 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:09:47.237156 (+ 153us) server_negotiation.cc:407] Connection header received
0507 02:09:47.237547 (+ 391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:09:47.237548 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:09:47.237557 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:09:47.237606 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:09:47.238697 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:47.239313 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:09:47.239972 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:47.240074 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:47.241887 (+ 1813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:09:47.241891 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:09:47.241891 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:09:47.241897 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:09:47.243135 (+ 1238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:09:47.243448 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:09:47.243449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:09:47.243450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:09:47.243470 (+ 20us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:09:47.243617 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:09:47.243618 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:09:47.243618 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:09:47.243702 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:09:47.243753 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0507 02:09:47.243963 (+ 210us) server_negotiation.cc:299] Negotiation successful
0507 02:09:47.244044 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20240507 02:09:47.244225 13831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:47.236893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:36547 (local address 127.31.131.65:58295)
0507 02:09:47.237058 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:47.237083 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0507 02:09:47.237473 (+ 390us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:09:47.237645 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:09:47.237648 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:09:47.237897 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:09:47.238617 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:47.238621 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:47.239394 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:09:47.239395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:09:47.239902 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:47.239904 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:47.240035 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:47.240516 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:09:47.240530 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:09:47.241809 (+ 1279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:09:47.243212 (+ 1403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:09:47.243217 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:09:47.243220 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:09:47.243389 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:09:47.243520 (+ 131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:09:47.243521 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:09:47.243521 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:09:47.243570 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:09:47.243752 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:09:47.243757 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:09:47.243855 (+ 98us) client_negotiation.cc:769] Sending connection context
0507 02:09:47.243939 (+ 84us) client_negotiation.cc:240] Negotiation successful
0507 02:09:47.244051 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":66,"threads_started":1}
I20240507 02:09:47.244628 13829 heartbeater.cc:347] Connected to a master server at 127.31.131.126:36547
I20240507 02:09:47.244732 13829 heartbeater.cc:464] Registering TS with master...
I20240507 02:09:47.244926 13829 heartbeater.cc:511] Master 127.31.131.126:36547 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.31.131.66@KRBTEST.COM; defaulting to no policy
I20240507 02:09:47.245362 13638 ts_manager.cc:194] Registered new tserver with Master: 1bec15577655406baa80dd159df72708 (127.31.131.65:33661)
I20240507 02:09:47.245954 13638 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.131.65@KRBTEST.COM'} at 127.31.131.65:58295
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab.
Entry for principal kudu/127.31.131.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab.
I20240507 02:09:47.258603 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.66:0
--local_ip_for_outbound_sockets=127.31.131.66
--webserver_interface=127.31.131.66
--webserver_port=0
--tserver_master_addrs=127.31.131.126:36547
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab
--principal=kudu/127.31.131.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:44929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:09:47.331830 13834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:09:47.331986 13834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:09:47.332005 13834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:09:47.333179 13834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:09:47.333211 13834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:09:47.333233 13834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.66
I20240507 02:09:47.334654 13834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:44929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab
--principal=kudu/127.31.131.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.131.66
--webserver_port=0
--tserver_master_addrs=127.31.131.126:36547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.66
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:09:47.334798 13834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:09:47.334930 13834 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:09:47.338420 13840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:09:47.338515 13834 server_base.cc:1034] running on GCE node
W20240507 02:09:47.338450 13839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:09:47.338608 13842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:09:47.338835 13834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:09:47.339021 13834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:09:47.340137 13834 hybrid_clock.cc:648] HybridClock initialized: now 1715047787340131 us; error 26 us; skew 500 ppm
May 07 02:09:47 dist-test-slave-v0bg krb5kdc[13593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047787, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:09:47.342120 13834 init.cc:377] Logged in from keytab as kudu/127.31.131.66@KRBTEST.COM (short username kudu)
I20240507 02:09:47.342480 13834 webserver.cc:460] Webserver started at http://127.31.131.66:37415/ using document root <none> and password file <none>
I20240507 02:09:47.342643 13834 fs_manager.cc:359] Metadata directory not provided
I20240507 02:09:47.342681 13834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:09:47.342790 13834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:09:47.343632 13834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-1/data/instance:
uuid: "3b07d0ce58504635ac42242bafc1719a"
format_stamp: "Formatted at 2024-05-07 02:09:47 on dist-test-slave-v0bg"
server_key: "3c13638d8c8706f0a6d9f32a285502e2"
server_key_iv: "57b4055fe2a88d95fd23cda7ed4ea520"
server_key_version: "encryptionkey@0"
I20240507 02:09:47.343909 13834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-1/wal/instance:
uuid: "3b07d0ce58504635ac42242bafc1719a"
format_stamp: "Formatted at 2024-05-07 02:09:47 on dist-test-slave-v0bg"
server_key: "3c13638d8c8706f0a6d9f32a285502e2"
server_key_iv: "57b4055fe2a88d95fd23cda7ed4ea520"
server_key_version: "encryptionkey@0"
I20240507 02:09:47.344870 13834 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240507 02:09:47.345414 13849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:09:47.345547 13834 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240507 02:09:47.345613 13834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-1/wal
uuid: "3b07d0ce58504635ac42242bafc1719a"
format_stamp: "Formatted at 2024-05-07 02:09:47 on dist-test-slave-v0bg"
server_key: "3c13638d8c8706f0a6d9f32a285502e2"
server_key_iv: "57b4055fe2a88d95fd23cda7ed4ea520"
server_key_version: "encryptionkey@0"
I20240507 02:09:47.345666 13834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:09:47.352639 13834 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:09:47.353930 13834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:09:47.354023 13834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:09:47.354175 13834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:09:47.354419 13834 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:09:47.354449 13834 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:09:47.354475 13834 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:09:47.354490 13834 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:09:47.359367 13834 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.66:35163
I20240507 02:09:47.359439 13962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.66:35163 every 8 connection(s)
I20240507 02:09:47.359761 13834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
I20240507 02:09:47.361670 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 13834
I20240507 02:09:47.361773 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-1/wal/instance
I20240507 02:09:47.361936 32269 external_mini_cluster.cc:1428] Setting key 163949a7a6ad2cda8cf3d900027f28c8
May 07 02:09:47 dist-test-slave-v0bg krb5kdc[13593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047787, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.66@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:09:47.367503 13696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:47.360326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.66:48833 (local address 127.31.131.126:36547)
0507 02:09:47.360379 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0507 02:09:47.360381 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:09:47.360510 (+ 129us) server_negotiation.cc:407] Connection header received
0507 02:09:47.360920 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:09:47.360921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:09:47.360932 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:09:47.360987 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:09:47.361930 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:47.362498 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:09:47.363178 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:47.363284 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:47.365124 (+ 1840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:09:47.365128 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:09:47.365128 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:09:47.365133 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:09:47.366379 (+ 1246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:09:47.366808 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:09:47.366809 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:09:47.366809 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:09:47.366828 (+ 19us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:09:47.367010 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:09:47.367011 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:09:47.367011 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:09:47.367081 (+ 70us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:09:47.367120 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0507 02:09:47.367383 (+ 263us) server_negotiation.cc:299] Negotiation successful
0507 02:09:47.367442 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20240507 02:09:47.367713 13965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:47.360303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:36547 (local address 127.31.131.66:48833)
0507 02:09:47.360423 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:47.360445 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0507 02:09:47.360832 (+ 387us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:09:47.361037 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:09:47.361040 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:09:47.361273 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:09:47.361846 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:47.361851 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:47.362606 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:09:47.362606 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:09:47.363110 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:47.363112 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:47.363236 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:47.363639 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:09:47.363654 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:09:47.365040 (+ 1386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:09:47.366453 (+ 1413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:09:47.366459 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:09:47.366461 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:09:47.366741 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:09:47.366880 (+ 139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:09:47.366881 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:09:47.366881 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:09:47.366953 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:09:47.367127 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:09:47.367131 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:09:47.367261 (+ 130us) client_negotiation.cc:769] Sending connection context
0507 02:09:47.367376 (+ 115us) client_negotiation.cc:240] Negotiation successful
0507 02:09:47.367496 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":63,"threads_started":1}
I20240507 02:09:47.368157 13963 heartbeater.cc:347] Connected to a master server at 127.31.131.126:36547
I20240507 02:09:47.368238 13963 heartbeater.cc:464] Registering TS with master...
I20240507 02:09:47.368394 13963 heartbeater.cc:511] Master 127.31.131.126:36547 requested a full tablet report, sending...
I20240507 02:09:47.368727 13638 ts_manager.cc:194] Registered new tserver with Master: 3b07d0ce58504635ac42242bafc1719a (127.31.131.66:35163)
I20240507 02:09:47.369063 13638 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.131.66@KRBTEST.COM'} at 127.31.131.66:48833
WARNING: no policy specified for kudu/127.31.131.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab.
Entry for principal kudu/127.31.131.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab.
I20240507 02:09:47.382894 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.67:0
--local_ip_for_outbound_sockets=127.31.131.67
--webserver_interface=127.31.131.67
--webserver_port=0
--tserver_master_addrs=127.31.131.126:36547
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab
--principal=kudu/127.31.131.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:44929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:09:47.454583 13968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:09:47.454778 13968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:09:47.454809 13968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:09:47.456000 13968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:09:47.456038 13968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:09:47.456070 13968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.67
I20240507 02:09:47.457433 13968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:44929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab
--principal=kudu/127.31.131.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.131.67
--webserver_port=0
--tserver_master_addrs=127.31.131.126:36547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.67
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:09:47.457612 13968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:09:47.457772 13968 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:09:47.461306 13973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:09:47.461318 13974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:09:47.461344 13968 server_base.cc:1034] running on GCE node
W20240507 02:09:47.461479 13976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:09:47.461625 13968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:09:47.461802 13968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:09:47.462915 13968 hybrid_clock.cc:648] HybridClock initialized: now 1715047787462901 us; error 32 us; skew 500 ppm
May 07 02:09:47 dist-test-slave-v0bg krb5kdc[13593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047787, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:09:47.464900 13968 init.cc:377] Logged in from keytab as kudu/127.31.131.67@KRBTEST.COM (short username kudu)
I20240507 02:09:47.465242 13968 webserver.cc:460] Webserver started at http://127.31.131.67:37037/ using document root <none> and password file <none>
I20240507 02:09:47.465375 13968 fs_manager.cc:359] Metadata directory not provided
I20240507 02:09:47.465406 13968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:09:47.465477 13968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:09:47.466274 13968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-2/data/instance:
uuid: "cb007fe5498b4f9b95aea33b51a36185"
format_stamp: "Formatted at 2024-05-07 02:09:47 on dist-test-slave-v0bg"
server_key: "a8aefd4f61d9cc8dc9f76d6cada64ca6"
server_key_iv: "cd7df357cfd495a12e0f9e36330fd9a1"
server_key_version: "encryptionkey@0"
I20240507 02:09:47.466507 13968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-2/wal/instance:
uuid: "cb007fe5498b4f9b95aea33b51a36185"
format_stamp: "Formatted at 2024-05-07 02:09:47 on dist-test-slave-v0bg"
server_key: "a8aefd4f61d9cc8dc9f76d6cada64ca6"
server_key_iv: "cd7df357cfd495a12e0f9e36330fd9a1"
server_key_version: "encryptionkey@0"
I20240507 02:09:47.467512 13968 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240507 02:09:47.468040 13983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:09:47.468168 13968 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240507 02:09:47.468238 13968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-2/wal
uuid: "cb007fe5498b4f9b95aea33b51a36185"
format_stamp: "Formatted at 2024-05-07 02:09:47 on dist-test-slave-v0bg"
server_key: "a8aefd4f61d9cc8dc9f76d6cada64ca6"
server_key_iv: "cd7df357cfd495a12e0f9e36330fd9a1"
server_key_version: "encryptionkey@0"
I20240507 02:09:47.468292 13968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:09:47.481298 13968 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:09:47.482838 13968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:09:47.482926 13968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:09:47.483081 13968 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:09:47.483331 13968 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:09:47.483361 13968 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:09:47.483386 13968 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:09:47.483415 13968 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:09:47.488114 13968 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.67:43055
I20240507 02:09:47.488163 14096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.67:43055 every 8 connection(s)
I20240507 02:09:47.488413 13968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
May 07 02:09:47 dist-test-slave-v0bg krb5kdc[13593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047787, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.67@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:09:47.495059 13696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:47.488981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.67:48201 (local address 127.31.131.126:36547)
0507 02:09:47.489024 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0507 02:09:47.489025 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:09:47.489168 (+ 143us) server_negotiation.cc:407] Connection header received
0507 02:09:47.489577 (+ 409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:09:47.489578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:09:47.489586 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:09:47.489636 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:09:47.490463 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:47.490961 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:09:47.491430 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:47.491528 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:47.493121 (+ 1593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:09:47.493125 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:09:47.493125 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:09:47.493131 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:09:47.494155 (+ 1024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:09:47.494449 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:09:47.494450 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:09:47.494450 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:09:47.494466 (+ 16us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:09:47.494627 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:09:47.494628 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:09:47.494628 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:09:47.494706 (+ 78us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:09:47.494754 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0507 02:09:47.494936 (+ 182us) server_negotiation.cc:299] Negotiation successful
0507 02:09:47.494995 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20240507 02:09:47.495144 14099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:47.488954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:36547 (local address 127.31.131.67:48201)
0507 02:09:47.489071 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:47.489098 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0507 02:09:47.489503 (+ 405us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:09:47.489684 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:09:47.489687 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:09:47.489939 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:09:47.490398 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:47.490401 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:47.491019 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:09:47.491020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:09:47.491375 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:47.491376 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:47.491467 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:47.491851 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:09:47.491862 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:09:47.493043 (+ 1181us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:09:47.494215 (+ 1172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:09:47.494219 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:09:47.494221 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:09:47.494398 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:09:47.494505 (+ 107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:09:47.494506 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:09:47.494506 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:09:47.494579 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:09:47.494750 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:09:47.494753 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:09:47.494844 (+ 91us) client_negotiation.cc:769] Sending connection context
0507 02:09:47.494919 (+ 75us) client_negotiation.cc:240] Negotiation successful
0507 02:09:47.495008 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":51,"threads_started":1}
I20240507 02:09:47.495565 14097 heartbeater.cc:347] Connected to a master server at 127.31.131.126:36547
I20240507 02:09:47.495640 14097 heartbeater.cc:464] Registering TS with master...
I20240507 02:09:47.495806 14097 heartbeater.cc:511] Master 127.31.131.126:36547 requested a full tablet report, sending...
I20240507 02:09:47.495944 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 13968
I20240507 02:09:47.496039 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-2/wal/instance
I20240507 02:09:47.496197 32269 external_mini_cluster.cc:1428] Setting key 8284d7654bf3e6a7e3dd4746878c668c
I20240507 02:09:47.496186 13638 ts_manager.cc:194] Registered new tserver with Master: cb007fe5498b4f9b95aea33b51a36185 (127.31.131.67:43055)
I20240507 02:09:47.496547 13638 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.131.67@KRBTEST.COM'} at 127.31.131.67:48201
I20240507 02:09:47.496994 32269 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240507 02:09:47.502609 13696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:47.498131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41850 (local address 127.31.131.126:36547)
0507 02:09:47.498179 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0507 02:09:47.498181 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:09:47.498263 (+ 82us) server_negotiation.cc:407] Connection header received
0507 02:09:47.498319 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:09:47.498320 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:09:47.498328 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:09:47.498370 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:09:47.499023 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:47.499464 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:09:47.499968 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:47.500113 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:47.500859 (+ 746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:09:47.500862 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:09:47.500862 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:09:47.500867 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:09:47.501785 (+ 918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:09:47.502030 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:09:47.502031 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:09:47.502031 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:09:47.502050 (+ 19us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:09:47.502171 (+ 121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:09:47.502172 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:09:47.502172 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:09:47.502268 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:09:47.502349 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0507 02:09:47.502447 (+ 98us) server_negotiation.cc:299] Negotiation successful
0507 02:09:47.502513 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20240507 02:09:47.503765 13638 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41850:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 02:09:47.505092 13638 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 02:09:47.509593 14107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:47.506822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.67:43055 (local address 127.0.0.1:40852)
0507 02:09:47.507053 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:47.507078 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0507 02:09:47.507144 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:09:47.507330 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:09:47.507333 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:09:47.507335 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:09:47.507561 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:47.507563 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:47.508609 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:09:47.508610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:09:47.509347 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:47.509349 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:47.509418 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:47.509449 (+ 31us) client_negotiation.cc:769] Sending connection context
0507 02:09:47.509495 (+ 46us) client_negotiation.cc:240] Negotiation successful
0507 02:09:47.509528 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":60,"threads_started":1}
I20240507 02:09:47.510095 14109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:47.507025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40852 (local address 127.31.131.67:43055)
0507 02:09:47.507170 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0507 02:09:47.507180 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0507 02:09:47.507189 (+ 9us) server_negotiation.cc:407] Connection header received
0507 02:09:47.507225 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:09:47.507226 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:09:47.507238 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:09:47.507312 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:09:47.507637 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:47.508539 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:09:47.509447 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:47.509918 (+ 471us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:47.509961 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0507 02:09:47.509990 (+ 29us) server_negotiation.cc:299] Negotiation successful
0507 02:09:47.510025 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":86,"threads_started":1}
I20240507 02:09:47.510437 14108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:47.506904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60124 (local address 127.31.131.65:33661)
0507 02:09:47.507048 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0507 02:09:47.507055 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0507 02:09:47.507312 (+ 257us) server_negotiation.cc:407] Connection header received
0507 02:09:47.507352 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:09:47.507354 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:09:47.507369 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:09:47.507433 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:09:47.507711 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:47.508814 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:09:47.509689 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:47.510145 (+ 456us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:47.510187 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0507 02:09:47.510321 (+ 134us) server_negotiation.cc:299] Negotiation successful
0507 02:09:47.510365 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":79,"threads_started":1}
I20240507 02:09:47.510643 14105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:47.506827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.65:33661 (local address 127.0.0.1:60124)
0507 02:09:47.507241 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:47.507249 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0507 02:09:47.507294 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:09:47.507481 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:09:47.507482 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:09:47.507484 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:09:47.507639 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:47.507642 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:47.508899 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:09:47.508900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:09:47.509576 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:47.509578 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:47.510225 (+ 647us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:47.510229 (+ 4us) client_negotiation.cc:769] Sending connection context
0507 02:09:47.510514 (+ 285us) client_negotiation.cc:240] Negotiation successful
0507 02:09:47.510583 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":78,"threads_started":1}
I20240507 02:09:47.510593 14031 tablet_service.cc:1467] Processing CreateTablet for tablet 19f197092e2143dcbbb862c1bffa15ab (DEFAULT_TABLE table=test-table [id=2374516b5af746449b7f83712e831b42]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:09:47.510794 14110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:47.507057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48268 (local address 127.31.131.66:35163)
0507 02:09:47.507198 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0507 02:09:47.507205 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0507 02:09:47.507412 (+ 207us) server_negotiation.cc:407] Connection header received
0507 02:09:47.507471 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:09:47.507473 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:09:47.507488 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:09:47.507553 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:09:47.507850 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:47.508919 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:09:47.510147 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:47.510607 (+ 460us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:47.510646 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0507 02:09:47.510673 (+ 27us) server_negotiation.cc:299] Negotiation successful
0507 02:09:47.510711 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":70,"threads_started":1}
I20240507 02:09:47.510870 14031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19f197092e2143dcbbb862c1bffa15ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:09:47.510936 14106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:47.506840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.66:35163 (local address 127.0.0.1:48268)
0507 02:09:47.507359 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:47.507366 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0507 02:09:47.507413 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:09:47.507555 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:09:47.507556 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:09:47.507558 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:09:47.507767 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:47.507769 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:47.508991 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:09:47.508992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:09:47.509825 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:47.509828 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:47.509885 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:47.509888 (+ 3us) client_negotiation.cc:769] Sending connection context
0507 02:09:47.509914 (+ 26us) client_negotiation.cc:240] Negotiation successful
0507 02:09:47.509946 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":73,"threads_started":1}
I20240507 02:09:47.511376 13897 tablet_service.cc:1467] Processing CreateTablet for tablet 19f197092e2143dcbbb862c1bffa15ab (DEFAULT_TABLE table=test-table [id=2374516b5af746449b7f83712e831b42]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:09:47.511592 13897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19f197092e2143dcbbb862c1bffa15ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:09:47.513103 14111 tablet_bootstrap.cc:492] T 19f197092e2143dcbbb862c1bffa15ab P cb007fe5498b4f9b95aea33b51a36185: Bootstrap starting.
I20240507 02:09:47.513401 13763 tablet_service.cc:1467] Processing CreateTablet for tablet 19f197092e2143dcbbb862c1bffa15ab (DEFAULT_TABLE table=test-table [id=2374516b5af746449b7f83712e831b42]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:09:47.513610 13763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19f197092e2143dcbbb862c1bffa15ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:09:47.513701 14111 tablet_bootstrap.cc:654] T 19f197092e2143dcbbb862c1bffa15ab P cb007fe5498b4f9b95aea33b51a36185: Neither blocks nor log segments found. Creating new log.
I20240507 02:09:47.513981 14111 log.cc:824] T 19f197092e2143dcbbb862c1bffa15ab P cb007fe5498b4f9b95aea33b51a36185: Log is configured to *not* fsync() on all Append() calls
I20240507 02:09:47.514228 14112 tablet_bootstrap.cc:492] T 19f197092e2143dcbbb862c1bffa15ab P 3b07d0ce58504635ac42242bafc1719a: Bootstrap starting.
I20240507 02:09:47.514708 14112 tablet_bootstrap.cc:654] T 19f197092e2143dcbbb862c1bffa15ab P 3b07d0ce58504635ac42242bafc1719a: Neither blocks nor log segments found. Creating new log.
I20240507 02:09:47.514973 14112 log.cc:824] T 19f197092e2143dcbbb862c1bffa15ab P 3b07d0ce58504635ac42242bafc1719a: Log is configured to *not* fsync() on all Append() calls
I20240507 02:09:47.515062 14111 tablet_bootstrap.cc:492] T 19f197092e2143dcbbb862c1bffa15ab P cb007fe5498b4f9b95aea33b51a36185: No bootstrap required, opened a new log
I20240507 02:09:47.515125 14111 ts_tablet_manager.cc:1367] T 19f197092e2143dcbbb862c1bffa15ab P cb007fe5498b4f9b95aea33b51a36185: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240507 02:09:47.515594 14112 tablet_bootstrap.cc:492] T 19f197092e2143dcbbb862c1bffa15ab P 3b07d0ce58504635ac42242bafc1719a: No bootstrap required, opened a new log
I20240507 02:09:47.515662 14112 ts_tablet_manager.cc:1367] T 19f197092e2143dcbbb862c1bffa15ab P 3b07d0ce58504635ac42242bafc1719a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240507 02:09:47.517181 14111 raft_consensus.cc:348] T 19f197092e2143dcbbb862c1bffa15ab P cb007fe5498b4f9b95aea33b51a36185 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb007fe5498b4f9b95aea33b51a36185" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 43055 } } peers { permanent_uuid: "1bec15577655406baa80dd159df72708" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33661 } } peers { permanent_uuid: "3b07d0ce58504635ac42242bafc1719a" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 35163 } }
I20240507 02:09:47.517326 14111 raft_consensus.cc:374] T 19f197092e2143dcbbb862c1bffa15ab P cb007fe5498b4f9b95aea33b51a36185 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:09:47.517350 14111 raft_consensus.cc:729] T 19f197092e2143dcbbb862c1bffa15ab P cb007fe5498b4f9b95aea33b51a36185 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb007fe5498b4f9b95aea33b51a36185, State: Initialized, Role: FOLLOWER
I20240507 02:09:47.517444 14111 consensus_queue.cc:260] T 19f197092e2143dcbbb862c1bffa15ab P cb007fe5498b4f9b95aea33b51a36185 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb007fe5498b4f9b95aea33b51a36185" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 43055 } } peers { permanent_uuid: "1bec15577655406baa80dd159df72708" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33661 } } peers { permanent_uuid: "3b07d0ce58504635ac42242bafc1719a" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 35163 } }
I20240507 02:09:47.517668 14111 ts_tablet_manager.cc:1398] T 19f197092e2143dcbbb862c1bffa15ab P cb007fe5498b4f9b95aea33b51a36185: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240507 02:09:47.517613 14112 raft_consensus.cc:348] T 19f197092e2143dcbbb862c1bffa15ab P 3b07d0ce58504635ac42242bafc1719a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb007fe5498b4f9b95aea33b51a36185" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 43055 } } peers { permanent_uuid: "1bec15577655406baa80dd159df72708" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33661 } } peers { permanent_uuid: "3b07d0ce58504635ac42242bafc1719a" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 35163 } }
I20240507 02:09:47.517699 14097 heartbeater.cc:503] Master 127.31.131.126:36547 was elected leader, sending a full tablet report...
I20240507 02:09:47.517716 14112 raft_consensus.cc:374] T 19f197092e2143dcbbb862c1bffa15ab P 3b07d0ce58504635ac42242bafc1719a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:09:47.517750 14112 raft_consensus.cc:729] T 19f197092e2143dcbbb862c1bffa15ab P 3b07d0ce58504635ac42242bafc1719a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b07d0ce58504635ac42242bafc1719a, State: Initialized, Role: FOLLOWER
I20240507 02:09:47.517828 14112 consensus_queue.cc:260] T 19f197092e2143dcbbb862c1bffa15ab P 3b07d0ce58504635ac42242bafc1719a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb007fe5498b4f9b95aea33b51a36185" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 43055 } } peers { permanent_uuid: "1bec15577655406baa80dd159df72708" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33661 } } peers { permanent_uuid: "3b07d0ce58504635ac42242bafc1719a" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 35163 } }
I20240507 02:09:47.518049 14112 ts_tablet_manager.cc:1398] T 19f197092e2143dcbbb862c1bffa15ab P 3b07d0ce58504635ac42242bafc1719a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240507 02:09:47.518077 13963 heartbeater.cc:503] Master 127.31.131.126:36547 was elected leader, sending a full tablet report...
I20240507 02:09:47.518219 14115 tablet_bootstrap.cc:492] T 19f197092e2143dcbbb862c1bffa15ab P 1bec15577655406baa80dd159df72708: Bootstrap starting.
I20240507 02:09:47.518780 14115 tablet_bootstrap.cc:654] T 19f197092e2143dcbbb862c1bffa15ab P 1bec15577655406baa80dd159df72708: Neither blocks nor log segments found. Creating new log.
I20240507 02:09:47.518729 14116 raft_consensus.cc:482] T 19f197092e2143dcbbb862c1bffa15ab P cb007fe5498b4f9b95aea33b51a36185 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 02:09:47.518852 14116 raft_consensus.cc:504] T 19f197092e2143dcbbb862c1bffa15ab P cb007fe5498b4f9b95aea33b51a36185 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb007fe5498b4f9b95aea33b51a36185" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 43055 } } peers { permanent_uuid: "1bec15577655406baa80dd159df72708" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33661 } } peers { permanent_uuid: "3b07d0ce58504635ac42242bafc1719a" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 35163 } }
I20240507 02:09:47.519055 14115 log.cc:824] T 19f197092e2143dcbbb862c1bffa15ab P 1bec15577655406baa80dd159df72708: Log is configured to *not* fsync() on all Append() calls
I20240507 02:09:47.519662 14116 leader_election.cc:288] T 19f197092e2143dcbbb862c1bffa15ab P cb007fe5498b4f9b95aea33b51a36185 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1bec15577655406baa80dd159df72708 (127.31.131.65:33661), 3b07d0ce58504635ac42242bafc1719a (127.31.131.66:35163)
I20240507 02:09:47.519699 14115 tablet_bootstrap.cc:492] T 19f197092e2143dcbbb862c1bffa15ab P 1bec15577655406baa80dd159df72708: No bootstrap required, opened a new log
I20240507 02:09:47.519785 14115 ts_tablet_manager.cc:1367] T 19f197092e2143dcbbb862c1bffa15ab P 1bec15577655406baa80dd159df72708: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240507 02:09:47.521236 14115 raft_consensus.cc:348] T 19f197092e2143dcbbb862c1bffa15ab P 1bec15577655406baa80dd159df72708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb007fe5498b4f9b95aea33b51a36185" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 43055 } } peers { permanent_uuid: "1bec15577655406baa80dd159df72708" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33661 } } peers { permanent_uuid: "3b07d0ce58504635ac42242bafc1719a" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 35163 } }
I20240507 02:09:47.521353 14115 raft_consensus.cc:374] T 19f197092e2143dcbbb862c1bffa15ab P 1bec15577655406baa80dd159df72708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:09:47.521355 14099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:47.519310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.66:35163 (local address 127.31.131.67:49019)
0507 02:09:47.519373 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:47.519382 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0507 02:09:47.519427 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:09:47.519557 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:09:47.519558 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:09:47.519560 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:09:47.519751 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:47.519752 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:47.520335 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:09:47.520336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:09:47.521146 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:47.521147 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:47.521236 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:47.521240 (+ 4us) client_negotiation.cc:769] Sending connection context
0507 02:09:47.521267 (+ 27us) client_negotiation.cc:240] Negotiation successful
0507 02:09:47.521297 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20240507 02:09:47.521375 14115 raft_consensus.cc:729] T 19f197092e2143dcbbb862c1bffa15ab P 1bec15577655406baa80dd159df72708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bec15577655406baa80dd159df72708, State: Initialized, Role: FOLLOWER
I20240507 02:09:47.521446 14115 consensus_queue.cc:260] T 19f197092e2143dcbbb862c1bffa15ab P 1bec15577655406baa80dd159df72708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb007fe5498b4f9b95aea33b51a36185" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 43055 } } peers { permanent_uuid: "1bec15577655406baa80dd159df72708" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33661 } } peers { permanent_uuid: "3b07d0ce58504635ac42242bafc1719a" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 35163 } }
I20240507 02:09:47.521636 14115 ts_tablet_manager.cc:1398] T 19f197092e2143dcbbb862c1bffa15ab P 1bec15577655406baa80dd159df72708: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240507 02:09:47.521684 13829 heartbeater.cc:503] Master 127.31.131.126:36547 was elected leader, sending a full tablet report...
I20240507 02:09:47.521790 14110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:47.519373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.67:49019 (local address 127.31.131.66:35163)
0507 02:09:47.519416 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0507 02:09:47.519418 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:09:47.519426 (+ 8us) server_negotiation.cc:407] Connection header received
0507 02:09:47.519485 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:09:47.519485 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:09:47.519492 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:09:47.519541 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:09:47.519808 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:47.520267 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:09:47.521231 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:47.521683 (+ 452us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:47.521692 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0507 02:09:47.521722 (+ 30us) server_negotiation.cc:299] Negotiation successful
0507 02:09:47.521751 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20240507 02:09:47.522171 13917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19f197092e2143dcbbb862c1bffa15ab" candidate_uuid: "cb007fe5498b4f9b95aea33b51a36185" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b07d0ce58504635ac42242bafc1719a" is_pre_election: true
I20240507 02:09:47.522258 13917 raft_consensus.cc:2454] T 19f197092e2143dcbbb862c1bffa15ab P 3b07d0ce58504635ac42242bafc1719a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb007fe5498b4f9b95aea33b51a36185 in term 0.
I20240507 02:09:47.522441 13987 leader_election.cc:302] T 19f197092e2143dcbbb862c1bffa15ab P cb007fe5498b4f9b95aea33b51a36185 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b07d0ce58504635ac42242bafc1719a, cb007fe5498b4f9b95aea33b51a36185; no voters:
I20240507 02:09:47.522565 14116 raft_consensus.cc:2781] T 19f197092e2143dcbbb862c1bffa15ab P cb007fe5498b4f9b95aea33b51a36185 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 02:09:47.522617 14116 raft_consensus.cc:482] T 19f197092e2143dcbbb862c1bffa15ab P cb007fe5498b4f9b95aea33b51a36185 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 02:09:47.522642 14116 raft_consensus.cc:3037] T 19f197092e2143dcbbb862c1bffa15ab P cb007fe5498b4f9b95aea33b51a36185 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:09:47.522992 14119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:47.519310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.65:33661 (local address 127.31.131.67:54873)
0507 02:09:47.519646 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:47.519655 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0507 02:09:47.519732 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:09:47.520438 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:09:47.520439 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:09:47.520440 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:09:47.520664 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:47.520665 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:47.521479 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:09:47.521480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:09:47.522797 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:47.522799 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:47.522858 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:47.522861 (+ 3us) client_negotiation.cc:769] Sending connection context
0507 02:09:47.522888 (+ 27us) client_negotiation.cc:240] Negotiation successful
0507 02:09:47.522923 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":208,"thread_start_us":81,"threads_started":1}
I20240507 02:09:47.523288 14116 raft_consensus.cc:504] T 19f197092e2143dcbbb862c1bffa15ab P cb007fe5498b4f9b95aea33b51a36185 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb007fe5498b4f9b95aea33b51a36185" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 43055 } } peers { permanent_uuid: "1bec15577655406baa80dd159df72708" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33661 } } peers { permanent_uuid: "3b07d0ce58504635ac42242bafc1719a" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 35163 } }
I20240507 02:09:47.523454 14116 leader_election.cc:288] T 19f197092e2143dcbbb862c1bffa15ab P cb007fe5498b4f9b95aea33b51a36185 [CANDIDATE]: Term 1 election: Requested vote from peers 1bec15577655406baa80dd159df72708 (127.31.131.65:33661), 3b07d0ce58504635ac42242bafc1719a (127.31.131.66:35163)
I20240507 02:09:47.523509 14108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:47.519499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.67:54873 (local address 127.31.131.65:33661)
0507 02:09:47.519645 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0507 02:09:47.519646 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:09:47.519703 (+ 57us) server_negotiation.cc:407] Connection header received
0507 02:09:47.520348 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:09:47.520348 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:09:47.520358 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:09:47.520384 (+ 26us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:09:47.520748 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:47.521412 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:09:47.523069 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:47.523411 (+ 342us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:47.523418 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0507 02:09:47.523436 (+ 18us) server_negotiation.cc:299] Negotiation successful
0507 02:09:47.523462 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20240507 02:09:47.523532 13917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19f197092e2143dcbbb862c1bffa15ab" candidate_uuid: "cb007fe5498b4f9b95aea33b51a36185" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b07d0ce58504635ac42242bafc1719a"
I20240507 02:09:47.523584 13917 raft_consensus.cc:3037] T 19f197092e2143dcbbb862c1bffa15ab P 3b07d0ce58504635ac42242bafc1719a [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:09:47.523725 13782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19f197092e2143dcbbb862c1bffa15ab" candidate_uuid: "cb007fe5498b4f9b95aea33b51a36185" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bec15577655406baa80dd159df72708"
I20240507 02:09:47.523721 13783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19f197092e2143dcbbb862c1bffa15ab" candidate_uuid: "cb007fe5498b4f9b95aea33b51a36185" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bec15577655406baa80dd159df72708" is_pre_election: true
I20240507 02:09:47.523819 13782 raft_consensus.cc:3037] T 19f197092e2143dcbbb862c1bffa15ab P 1bec15577655406baa80dd159df72708 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:09:47.524324 13917 raft_consensus.cc:2454] T 19f197092e2143dcbbb862c1bffa15ab P 3b07d0ce58504635ac42242bafc1719a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb007fe5498b4f9b95aea33b51a36185 in term 1.
I20240507 02:09:47.524545 13782 raft_consensus.cc:2454] T 19f197092e2143dcbbb862c1bffa15ab P 1bec15577655406baa80dd159df72708 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb007fe5498b4f9b95aea33b51a36185 in term 1.
I20240507 02:09:47.524578 13987 leader_election.cc:302] T 19f197092e2143dcbbb862c1bffa15ab P cb007fe5498b4f9b95aea33b51a36185 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b07d0ce58504635ac42242bafc1719a, cb007fe5498b4f9b95aea33b51a36185; no voters:
I20240507 02:09:47.524650 14116 raft_consensus.cc:2781] T 19f197092e2143dcbbb862c1bffa15ab P cb007fe5498b4f9b95aea33b51a36185 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:09:47.524793 14116 raft_consensus.cc:686] T 19f197092e2143dcbbb862c1bffa15ab P cb007fe5498b4f9b95aea33b51a36185 [term 1 LEADER]: Becoming Leader. State: Replica: cb007fe5498b4f9b95aea33b51a36185, State: Running, Role: LEADER
I20240507 02:09:47.524861 14116 consensus_queue.cc:237] T 19f197092e2143dcbbb862c1bffa15ab P cb007fe5498b4f9b95aea33b51a36185 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb007fe5498b4f9b95aea33b51a36185" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 43055 } } peers { permanent_uuid: "1bec15577655406baa80dd159df72708" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33661 } } peers { permanent_uuid: "3b07d0ce58504635ac42242bafc1719a" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 35163 } }
I20240507 02:09:47.525444 13638 catalog_manager.cc:5484] T 19f197092e2143dcbbb862c1bffa15ab P cb007fe5498b4f9b95aea33b51a36185 reported cstate change: term changed from 0 to 1, leader changed from <none> to cb007fe5498b4f9b95aea33b51a36185 (127.31.131.67). New cstate: current_term: 1 leader_uuid: "cb007fe5498b4f9b95aea33b51a36185" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb007fe5498b4f9b95aea33b51a36185" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 43055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bec15577655406baa80dd159df72708" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b07d0ce58504635ac42242bafc1719a" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 35163 } health_report { overall_health: UNKNOWN } } }
I20240507 02:09:47.531242 14109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:47.529469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40856 (local address 127.31.131.67:43055)
0507 02:09:47.529513 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0507 02:09:47.529515 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:09:47.529520 (+ 5us) server_negotiation.cc:407] Connection header received
0507 02:09:47.529544 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:09:47.529544 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:09:47.529556 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:09:47.529614 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0507 02:09:47.529839 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:47.530257 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:09:47.530813 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:47.530934 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:47.530963 (+ 29us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0507 02:09:47.531034 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0507 02:09:47.531095 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0507 02:09:47.531144 (+ 49us) server_negotiation.cc:299] Negotiation successful
0507 02:09:47.531163 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20240507 02:09:47.532824 13917 raft_consensus.cc:1261] T 19f197092e2143dcbbb862c1bffa15ab P 3b07d0ce58504635ac42242bafc1719a [term 1 FOLLOWER]: Refusing update from remote peer cb007fe5498b4f9b95aea33b51a36185: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 02:09:47.533111 14121 consensus_queue.cc:1035] T 19f197092e2143dcbbb862c1bffa15ab P cb007fe5498b4f9b95aea33b51a36185 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b07d0ce58504635ac42242bafc1719a" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 35163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 02:09:47.534747 13782 raft_consensus.cc:1261] T 19f197092e2143dcbbb862c1bffa15ab P 1bec15577655406baa80dd159df72708 [term 1 FOLLOWER]: Refusing update from remote peer cb007fe5498b4f9b95aea33b51a36185: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 02:09:47.535403 14125 mvcc.cc:205] Tried to move back new op lower bound from 7024835737732251648 to 7024835737702383616. Current Snapshot: MvccSnapshot[applied={T|T < 7024835737732251648}]
I20240507 02:09:47.537369 14121 consensus_queue.cc:1035] T 19f197092e2143dcbbb862c1bffa15ab P cb007fe5498b4f9b95aea33b51a36185 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bec15577655406baa80dd159df72708" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 02:09:47.537588 13637 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41850:
table { table_name: "test-table" } modify_external_catalogs: true
I20240507 02:09:47.537676 13637 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41850:
table { table_name: "test-table" } modify_external_catalogs: true
I20240507 02:09:47.539121 13637 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b3f5a6f84b18408c94e6a9c6e49ce384: Sending DeleteTablet for 3 replicas of tablet 19f197092e2143dcbbb862c1bffa15ab
I20240507 02:09:47.539436 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 13700
I20240507 02:09:47.539582 14031 tablet_service.cc:1514] Processing DeleteTablet for tablet 19f197092e2143dcbbb862c1bffa15ab with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:09:47 UTC) from {username='kudu', principal='kudu/127.31.131.126@KRBTEST.COM'} at 127.0.0.1:40852
I20240507 02:09:47.539973 13897 tablet_service.cc:1514] Processing DeleteTablet for tablet 19f197092e2143dcbbb862c1bffa15ab with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:09:47 UTC) from {username='kudu', principal='kudu/127.31.131.126@KRBTEST.COM'} at 127.0.0.1:48268
I20240507 02:09:47.540167 14134 tablet_replica.cc:330] T 19f197092e2143dcbbb862c1bffa15ab P 3b07d0ce58504635ac42242bafc1719a: stopping tablet replica
I20240507 02:09:47.540280 14134 raft_consensus.cc:2229] T 19f197092e2143dcbbb862c1bffa15ab P 3b07d0ce58504635ac42242bafc1719a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 02:09:47.540400 14134 raft_consensus.cc:2258] T 19f197092e2143dcbbb862c1bffa15ab P 3b07d0ce58504635ac42242bafc1719a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 02:09:47.540396 14133 tablet_replica.cc:330] T 19f197092e2143dcbbb862c1bffa15ab P cb007fe5498b4f9b95aea33b51a36185: stopping tablet replica
I20240507 02:09:47.540467 14133 raft_consensus.cc:2229] T 19f197092e2143dcbbb862c1bffa15ab P cb007fe5498b4f9b95aea33b51a36185 [term 1 LEADER]: Raft consensus shutting down.
I20240507 02:09:47.540648 14133 raft_consensus.cc:2258] T 19f197092e2143dcbbb862c1bffa15ab P cb007fe5498b4f9b95aea33b51a36185 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 02:09:47.540982 14133 ts_tablet_manager.cc:1875] T 19f197092e2143dcbbb862c1bffa15ab P cb007fe5498b4f9b95aea33b51a36185: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 02:09:47.542033 14134 ts_tablet_manager.cc:1875] T 19f197092e2143dcbbb862c1bffa15ab P 3b07d0ce58504635ac42242bafc1719a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 02:09:47.542431 14133 ts_tablet_manager.cc:1888] T 19f197092e2143dcbbb862c1bffa15ab P cb007fe5498b4f9b95aea33b51a36185: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240507 02:09:47.542498 14133 log.cc:1196] T 19f197092e2143dcbbb862c1bffa15ab P cb007fe5498b4f9b95aea33b51a36185: Deleting WAL directory at /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-2/wal/wals/19f197092e2143dcbbb862c1bffa15ab
I20240507 02:09:47.542793 14133 ts_tablet_manager.cc:1909] T 19f197092e2143dcbbb862c1bffa15ab P cb007fe5498b4f9b95aea33b51a36185: Deleting consensus metadata
I20240507 02:09:47.543119 13622 catalog_manager.cc:4830] TS cb007fe5498b4f9b95aea33b51a36185 (127.31.131.67:43055): tablet 19f197092e2143dcbbb862c1bffa15ab (table test-table [id=2374516b5af746449b7f83712e831b42]) successfully deleted
I20240507 02:09:47.543174 14134 ts_tablet_manager.cc:1888] T 19f197092e2143dcbbb862c1bffa15ab P 3b07d0ce58504635ac42242bafc1719a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240507 02:09:47.543233 14134 log.cc:1196] T 19f197092e2143dcbbb862c1bffa15ab P 3b07d0ce58504635ac42242bafc1719a: Deleting WAL directory at /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-1/wal/wals/19f197092e2143dcbbb862c1bffa15ab
I20240507 02:09:47.543447 14134 ts_tablet_manager.cc:1909] T 19f197092e2143dcbbb862c1bffa15ab P 3b07d0ce58504635ac42242bafc1719a: Deleting consensus metadata
I20240507 02:09:47.543745 13625 catalog_manager.cc:4830] TS 3b07d0ce58504635ac42242bafc1719a (127.31.131.66:35163): tablet 19f197092e2143dcbbb862c1bffa15ab (table test-table [id=2374516b5af746449b7f83712e831b42]) successfully deleted
W20240507 02:09:47.550025 13624 connection.cc:535] server connection from 127.31.131.65:58295 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240507 02:09:47.550029 13985 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.131.65:33661 (error 108)
W20240507 02:09:47.550179 13623 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.131.65:33661 (error 108)
I20240507 02:09:47.550297 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 13834
I20240507 02:09:47.550525 14119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:47.550421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.65:33661 (local address 127.31.131.67:55521)
0507 02:09:47.550468 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:47.550493 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.65:33661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20240507 02:09:47.551849 14106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:47.551737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.65:33661 (local address 127.0.0.1:60134)
0507 02:09:47.551787 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:47.551818 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.65:33661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20240507 02:09:47.551939 13623 catalog_manager.cc:4573] TS 1bec15577655406baa80dd159df72708 (127.31.131.65:33661): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 19f197092e2143dcbbb862c1bffa15ab: Network error: Client connection negotiation failed: client connection to 127.31.131.65:33661: connect: Connection refused (error 111)
I20240507 02:09:47.554865 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 13968
I20240507 02:09:47.558928 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 13606
I20240507 02:09:47.563397 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:36547
--webserver_interface=127.31.131.126
--webserver_port=40679
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab
--principal=kudu/127.31.131.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:44929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.131.126:36547
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:09:47.639227 14137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:09:47.639410 14137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:09:47.639437 14137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:09:47.640661 14137 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240507 02:09:47.640707 14137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:09:47.640724 14137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:09:47.640738 14137 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240507 02:09:47.640753 14137 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240507 02:09:47.642187 14137 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:44929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.31.131.126:36547
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab
--principal=kudu/127.31.131.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:36547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.131.126
--webserver_port=40679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:09:47.642393 14137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:09:47.642577 14137 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:09:47.649057 14144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:09:47.649078 14137 server_base.cc:1034] running on GCE node
W20240507 02:09:47.649045 14146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:09:47.649046 14143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:09:47.649425 14137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:09:47.649641 14137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:09:47.650758 14137 hybrid_clock.cc:648] HybridClock initialized: now 1715047787650748 us; error 28 us; skew 500 ppm
May 07 02:09:47 dist-test-slave-v0bg krb5kdc[13593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047787, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:09:47.652916 14137 init.cc:377] Logged in from keytab as kudu/127.31.131.126@KRBTEST.COM (short username kudu)
I20240507 02:09:47.653301 14137 webserver.cc:460] Webserver started at http://127.31.131.126:40679/ using document root <none> and password file <none>
I20240507 02:09:47.653471 14137 fs_manager.cc:359] Metadata directory not provided
I20240507 02:09:47.653520 14137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:09:47.654745 14137 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20240507 02:09:47.655364 14153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:09:47.655519 14137 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240507 02:09:47.655596 14137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/master-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/master-0/wal
uuid: "b3f5a6f84b18408c94e6a9c6e49ce384"
format_stamp: "Formatted at 2024-05-07 02:09:47 on dist-test-slave-v0bg"
server_key: "784f36bbe8cea968188f2cc04d2c7033"
server_key_iv: "8af29a7f465f9744faff075c642b1967"
server_key_version: "encryptionkey@0"
I20240507 02:09:47.655918 14137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:09:47.690186 14137 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:09:47.691948 14137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:09:47.692061 14137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:09:47.696070 14137 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.126:36547
I20240507 02:09:47.696110 14205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.126:36547 every 8 connection(s)
I20240507 02:09:47.696450 14137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
I20240507 02:09:47.697353 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 14137
I20240507 02:09:47.697597 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.65:33661
--local_ip_for_outbound_sockets=127.31.131.65
--tserver_master_addrs=127.31.131.126:36547
--webserver_port=41741
--webserver_interface=127.31.131.65
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab
--principal=kudu/127.31.131.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:44929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240507 02:09:47.697974 14206 sys_catalog.cc:263] Verifying existing consensus state
I20240507 02:09:47.698571 14206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3f5a6f84b18408c94e6a9c6e49ce384: Bootstrap starting.
I20240507 02:09:47.707543 14206 log.cc:824] T 00000000000000000000000000000000 P b3f5a6f84b18408c94e6a9c6e49ce384: Log is configured to *not* fsync() on all Append() calls
I20240507 02:09:47.709245 14206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3f5a6f84b18408c94e6a9c6e49ce384: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240507 02:09:47.709482 14206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3f5a6f84b18408c94e6a9c6e49ce384: Bootstrap complete.
I20240507 02:09:47.711442 14206 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3f5a6f84b18408c94e6a9c6e49ce384 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3f5a6f84b18408c94e6a9c6e49ce384" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 36547 } }
I20240507 02:09:47.711629 14206 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3f5a6f84b18408c94e6a9c6e49ce384 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3f5a6f84b18408c94e6a9c6e49ce384, State: Initialized, Role: FOLLOWER
I20240507 02:09:47.711755 14206 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3f5a6f84b18408c94e6a9c6e49ce384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3f5a6f84b18408c94e6a9c6e49ce384" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 36547 } }
I20240507 02:09:47.711822 14206 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3f5a6f84b18408c94e6a9c6e49ce384 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 02:09:47.711864 14206 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3f5a6f84b18408c94e6a9c6e49ce384 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 02:09:47.711908 14206 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3f5a6f84b18408c94e6a9c6e49ce384 [term 1 FOLLOWER]: Advancing to term 2
I20240507 02:09:47.712706 14206 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3f5a6f84b18408c94e6a9c6e49ce384 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3f5a6f84b18408c94e6a9c6e49ce384" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 36547 } }
I20240507 02:09:47.712795 14206 leader_election.cc:302] T 00000000000000000000000000000000 P b3f5a6f84b18408c94e6a9c6e49ce384 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3f5a6f84b18408c94e6a9c6e49ce384; no voters:
I20240507 02:09:47.712931 14206 leader_election.cc:288] T 00000000000000000000000000000000 P b3f5a6f84b18408c94e6a9c6e49ce384 [CANDIDATE]: Term 2 election: Requested vote from peers
I20240507 02:09:47.712977 14210 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3f5a6f84b18408c94e6a9c6e49ce384 [term 2 FOLLOWER]: Leader election won for term 2
I20240507 02:09:47.713079 14210 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3f5a6f84b18408c94e6a9c6e49ce384 [term 2 LEADER]: Becoming Leader. State: Replica: b3f5a6f84b18408c94e6a9c6e49ce384, State: Running, Role: LEADER
I20240507 02:09:47.713151 14206 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3f5a6f84b18408c94e6a9c6e49ce384 [sys.catalog]: configured and running, proceeding with master startup.
I20240507 02:09:47.713147 14210 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3f5a6f84b18408c94e6a9c6e49ce384 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3f5a6f84b18408c94e6a9c6e49ce384" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 36547 } }
I20240507 02:09:47.713960 14212 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3f5a6f84b18408c94e6a9c6e49ce384 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3f5a6f84b18408c94e6a9c6e49ce384. Latest consensus state: current_term: 2 leader_uuid: "b3f5a6f84b18408c94e6a9c6e49ce384" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3f5a6f84b18408c94e6a9c6e49ce384" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 36547 } } }
I20240507 02:09:47.714038 14212 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3f5a6f84b18408c94e6a9c6e49ce384 [sys.catalog]: This master's current role is: LEADER
I20240507 02:09:47.713943 14211 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3f5a6f84b18408c94e6a9c6e49ce384 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b3f5a6f84b18408c94e6a9c6e49ce384" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3f5a6f84b18408c94e6a9c6e49ce384" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 36547 } } }
I20240507 02:09:47.714190 14211 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3f5a6f84b18408c94e6a9c6e49ce384 [sys.catalog]: This master's current role is: LEADER
I20240507 02:09:47.714269 14220 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 02:09:47.714860 14220 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240507 02:09:47.714944 14220 catalog_manager.cc:1259] Loaded cluster ID: a212f29e4e124c83a118b6754b8cae85
I20240507 02:09:47.714973 14220 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240507 02:09:47.715170 14220 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20240507 02:09:47.716668 14220 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20240507 02:09:47.718257 14220 catalog_manager.cc:1504] Loading token signing keys...
I20240507 02:09:47.718402 14220 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3f5a6f84b18408c94e6a9c6e49ce384: Loaded TSK: 0
I20240507 02:09:47.718596 14220 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20240507 02:09:47.785518 14208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:09:47.785684 14208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:09:47.785703 14208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:09:47.786979 14208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:09:47.787019 14208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:09:47.787040 14208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.65
I20240507 02:09:47.788439 14208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:44929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab
--principal=kudu/127.31.131.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.65:33661
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.131.65
--webserver_port=41741
--tserver_master_addrs=127.31.131.126:36547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.65
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:09:47.788625 14208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:09:47.788808 14208 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:09:47.793040 14233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:09:47.793030 14234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:09:47.793164 14208 server_base.cc:1034] running on GCE node
W20240507 02:09:47.793030 14236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:09:47.793426 14208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:09:47.793619 14208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:09:47.794744 14208 hybrid_clock.cc:648] HybridClock initialized: now 1715047787794729 us; error 35 us; skew 500 ppm
May 07 02:09:47 dist-test-slave-v0bg krb5kdc[13593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047787, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:09:47.797037 14208 init.cc:377] Logged in from keytab as kudu/127.31.131.65@KRBTEST.COM (short username kudu)
I20240507 02:09:47.797412 14208 webserver.cc:460] Webserver started at http://127.31.131.65:41741/ using document root <none> and password file <none>
I20240507 02:09:47.797583 14208 fs_manager.cc:359] Metadata directory not provided
I20240507 02:09:47.797643 14208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:09:47.798897 14208 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20240507 02:09:47.799443 14243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:09:47.799587 14208 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240507 02:09:47.799654 14208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-0/wal
uuid: "1bec15577655406baa80dd159df72708"
format_stamp: "Formatted at 2024-05-07 02:09:47 on dist-test-slave-v0bg"
server_key: "8632f4ad1e8414ee6fb17ee667ac2f6e"
server_key_iv: "ec4f18d4430225476621a09832fcd7ea"
server_key_version: "encryptionkey@0"
I20240507 02:09:47.799921 14208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:09:47.813308 14208 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:09:47.815349 14208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:09:47.815459 14208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:09:47.815634 14208 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:09:47.816013 14250 ts_tablet_manager.cc:516] Loading tablet metadata (0/1 complete)
I20240507 02:09:47.817109 14208 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240507 02:09:47.817149 14208 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20240507 02:09:47.817180 14208 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240507 02:09:47.817704 14208 ts_tablet_manager.cc:590] Registered 1 tablets
I20240507 02:09:47.817735 14208 ts_tablet_manager.cc:569] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20240507 02:09:47.817832 14250 tablet_bootstrap.cc:492] T 19f197092e2143dcbbb862c1bffa15ab P 1bec15577655406baa80dd159df72708: Bootstrap starting.
I20240507 02:09:47.827558 14208 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.65:33661
I20240507 02:09:47.827896 14208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
I20240507 02:09:47.829087 14250 log.cc:824] T 19f197092e2143dcbbb862c1bffa15ab P 1bec15577655406baa80dd159df72708: Log is configured to *not* fsync() on all Append() calls
I20240507 02:09:47.829815 14250 tablet_bootstrap.cc:492] T 19f197092e2143dcbbb862c1bffa15ab P 1bec15577655406baa80dd159df72708: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240507 02:09:47.830026 14250 tablet_bootstrap.cc:492] T 19f197092e2143dcbbb862c1bffa15ab P 1bec15577655406baa80dd159df72708: Bootstrap complete.
I20240507 02:09:47.830039 14357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.65:33661 every 8 connection(s)
I20240507 02:09:47.831009 14250 ts_tablet_manager.cc:1367] T 19f197092e2143dcbbb862c1bffa15ab P 1bec15577655406baa80dd159df72708: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.009s
I20240507 02:09:47.831943 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 14208
I20240507 02:09:47.832204 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.66:35163
--local_ip_for_outbound_sockets=127.31.131.66
--tserver_master_addrs=127.31.131.126:36547
--webserver_port=37415
--webserver_interface=127.31.131.66
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab
--principal=kudu/127.31.131.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:44929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240507 02:09:47.832461 14250 raft_consensus.cc:348] T 19f197092e2143dcbbb862c1bffa15ab P 1bec15577655406baa80dd159df72708 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb007fe5498b4f9b95aea33b51a36185" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 43055 } } peers { permanent_uuid: "1bec15577655406baa80dd159df72708" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33661 } } peers { permanent_uuid: "3b07d0ce58504635ac42242bafc1719a" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 35163 } }
I20240507 02:09:47.832708 14250 raft_consensus.cc:729] T 19f197092e2143dcbbb862c1bffa15ab P 1bec15577655406baa80dd159df72708 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bec15577655406baa80dd159df72708, State: Initialized, Role: FOLLOWER
I20240507 02:09:47.832808 14250 consensus_queue.cc:260] T 19f197092e2143dcbbb862c1bffa15ab P 1bec15577655406baa80dd159df72708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb007fe5498b4f9b95aea33b51a36185" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 43055 } } peers { permanent_uuid: "1bec15577655406baa80dd159df72708" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33661 } } peers { permanent_uuid: "3b07d0ce58504635ac42242bafc1719a" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 35163 } }
I20240507 02:09:47.833020 14250 ts_tablet_manager.cc:1398] T 19f197092e2143dcbbb862c1bffa15ab P 1bec15577655406baa80dd159df72708: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
May 07 02:09:47 dist-test-slave-v0bg krb5kdc[13593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047787, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.65@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:09:47.840934 14360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:47.828590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:36547 (local address 127.31.131.65:35021)
0507 02:09:47.828769 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:47.828797 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0507 02:09:47.829278 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:09:47.829819 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:09:47.829823 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:09:47.830360 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:09:47.830748 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:47.830753 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:47.832271 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:09:47.832272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:09:47.832904 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:47.832907 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:47.833409 (+ 502us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:47.834018 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:09:47.834039 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:09:47.837980 (+ 3941us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:09:47.839730 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:09:47.839736 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:09:47.839738 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:09:47.839920 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:09:47.840089 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:09:47.840090 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:09:47.840090 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:09:47.840143 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:09:47.840424 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:09:47.840429 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:09:47.840531 (+ 102us) client_negotiation.cc:769] Sending connection context
0507 02:09:47.840618 (+ 87us) client_negotiation.cc:240] Negotiation successful
0507 02:09:47.840757 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":101,"threads_started":1}
I20240507 02:09:47.840947 14361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:47.828712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:35021 (local address 127.31.131.126:36547)
0507 02:09:47.829388 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0507 02:09:47.829400 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0507 02:09:47.829430 (+ 30us) server_negotiation.cc:407] Connection header received
0507 02:09:47.829563 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:09:47.829568 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:09:47.829617 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:09:47.829788 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:09:47.830888 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:47.832138 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:09:47.833013 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:47.833256 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:47.838084 (+ 4828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:09:47.838092 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:09:47.838092 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:09:47.838107 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:09:47.839646 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:09:47.839989 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:09:47.839992 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:09:47.839993 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:09:47.840036 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:09:47.840201 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:09:47.840202 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:09:47.840202 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:09:47.840367 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:09:47.840438 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0507 02:09:47.840683 (+ 245us) server_negotiation.cc:299] Negotiation successful
0507 02:09:47.840790 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":82,"threads_started":1}
I20240507 02:09:47.842365 14358 heartbeater.cc:347] Connected to a master server at 127.31.131.126:36547
I20240507 02:09:47.842458 14358 heartbeater.cc:464] Registering TS with master...
I20240507 02:09:47.842679 14358 heartbeater.cc:511] Master 127.31.131.126:36547 requested a full tablet report, sending...
I20240507 02:09:47.843340 14170 ts_manager.cc:194] Registered new tserver with Master: 1bec15577655406baa80dd159df72708 (127.31.131.65:33661)
I20240507 02:09:47.844358 14170 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.131.65@KRBTEST.COM'} at 127.31.131.65:35021
I20240507 02:09:47.845299 14358 heartbeater.cc:503] Master 127.31.131.126:36547 was elected leader, sending a full tablet report...
I20240507 02:09:47.847723 14365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:47.843865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.65:33661 (local address 127.0.0.1:60144)
0507 02:09:47.844856 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:47.844879 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0507 02:09:47.844952 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:09:47.845318 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:09:47.845321 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:09:47.845323 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:09:47.845988 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:47.845990 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:47.846905 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:09:47.846906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:09:47.847472 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:47.847473 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:47.847533 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:47.847560 (+ 27us) client_negotiation.cc:769] Sending connection context
0507 02:09:47.847607 (+ 47us) client_negotiation.cc:240] Negotiation successful
0507 02:09:47.847642 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":966,"thread_start_us":102,"threads_started":1}
I20240507 02:09:47.848230 14366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:47.844582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60144 (local address 127.31.131.65:33661)
0507 02:09:47.844733 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0507 02:09:47.844740 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0507 02:09:47.845019 (+ 279us) server_negotiation.cc:407] Connection header received
0507 02:09:47.845049 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:09:47.845051 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:09:47.845172 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:09:47.845249 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:09:47.846062 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:47.846833 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:09:47.847544 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:47.848105 (+ 561us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:47.848137 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0507 02:09:47.848162 (+ 25us) server_negotiation.cc:299] Negotiation successful
0507 02:09:47.848188 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":77,"threads_started":1}
I20240507 02:09:47.848484 14292 tablet_service.cc:1514] Processing DeleteTablet for tablet 19f197092e2143dcbbb862c1bffa15ab with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:09:47 UTC) from {username='kudu', principal='kudu/127.31.131.126@KRBTEST.COM'} at 127.0.0.1:60144
I20240507 02:09:47.848913 14367 tablet_replica.cc:330] stopping tablet replica
I20240507 02:09:47.849001 14367 raft_consensus.cc:2229] T 19f197092e2143dcbbb862c1bffa15ab P 1bec15577655406baa80dd159df72708 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 02:09:47.849069 14367 raft_consensus.cc:2258] T 19f197092e2143dcbbb862c1bffa15ab P 1bec15577655406baa80dd159df72708 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 02:09:47.849323 14367 ts_tablet_manager.cc:1875] T 19f197092e2143dcbbb862c1bffa15ab P 1bec15577655406baa80dd159df72708: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 02:09:47.851006 14367 ts_tablet_manager.cc:1888] T 19f197092e2143dcbbb862c1bffa15ab P 1bec15577655406baa80dd159df72708: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 02:09:47.851064 14367 log.cc:1196] T 19f197092e2143dcbbb862c1bffa15ab P 1bec15577655406baa80dd159df72708: Deleting WAL directory at /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-0/wal/wals/19f197092e2143dcbbb862c1bffa15ab
I20240507 02:09:47.851229 14367 ts_tablet_manager.cc:1909] T 19f197092e2143dcbbb862c1bffa15ab P 1bec15577655406baa80dd159df72708: Deleting consensus metadata
I20240507 02:09:47.851620 14155 catalog_manager.cc:4830] TS 1bec15577655406baa80dd159df72708 (127.31.131.65:33661): tablet 19f197092e2143dcbbb862c1bffa15ab (table test-table [id=2374516b5af746449b7f83712e831b42]) successfully deleted
W20240507 02:09:47.928246 14364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:09:47.928444 14364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:09:47.928469 14364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:09:47.929770 14364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:09:47.929821 14364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:09:47.929852 14364 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.66
I20240507 02:09:47.931401 14364 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:44929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab
--principal=kudu/127.31.131.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.66:35163
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.131.66
--webserver_port=37415
--tserver_master_addrs=127.31.131.126:36547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14364
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.66
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:09:47.931622 14364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:09:47.931821 14364 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240507 02:09:47.935721 14364 server_base.cc:1034] running on GCE node
W20240507 02:09:47.935758 14373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:09:47.935858 14372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:09:47.935865 14375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:09:47.936072 14364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:09:47.936298 14364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:09:47.937444 14364 hybrid_clock.cc:648] HybridClock initialized: now 1715047787937429 us; error 32 us; skew 500 ppm
May 07 02:09:47 dist-test-slave-v0bg krb5kdc[13593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047787, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:09:47.939838 14364 init.cc:377] Logged in from keytab as kudu/127.31.131.66@KRBTEST.COM (short username kudu)
I20240507 02:09:47.940234 14364 webserver.cc:460] Webserver started at http://127.31.131.66:37415/ using document root <none> and password file <none>
I20240507 02:09:47.940404 14364 fs_manager.cc:359] Metadata directory not provided
I20240507 02:09:47.940459 14364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:09:47.941686 14364 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20240507 02:09:47.942179 14382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:09:47.942312 14364 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240507 02:09:47.942389 14364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-1/wal
uuid: "3b07d0ce58504635ac42242bafc1719a"
format_stamp: "Formatted at 2024-05-07 02:09:47 on dist-test-slave-v0bg"
server_key: "3c13638d8c8706f0a6d9f32a285502e2"
server_key_iv: "57b4055fe2a88d95fd23cda7ed4ea520"
server_key_version: "encryptionkey@0"
I20240507 02:09:47.942741 14364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:09:47.952963 14364 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:09:47.955020 14364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:09:47.955142 14364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:09:47.955335 14364 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:09:47.955688 14364 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:09:47.955739 14364 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:09:47.955770 14364 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:09:47.955787 14364 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:09:47.961357 14364 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.66:35163
I20240507 02:09:47.961412 14495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.66:35163 every 8 connection(s)
I20240507 02:09:47.961699 14364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
May 07 02:09:47 dist-test-slave-v0bg krb5kdc[13593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047787, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.66@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:09:47.967612 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 14364
I20240507 02:09:47.967875 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.67:43055
--local_ip_for_outbound_sockets=127.31.131.67
--tserver_master_addrs=127.31.131.126:36547
--webserver_port=37037
--webserver_interface=127.31.131.67
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab
--principal=kudu/127.31.131.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:44929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240507 02:09:47.969661 14361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:47.962313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.66:53235 (local address 127.31.131.126:36547)
0507 02:09:47.962363 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0507 02:09:47.962365 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:09:47.962523 (+ 158us) server_negotiation.cc:407] Connection header received
0507 02:09:47.963023 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:09:47.963024 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:09:47.963034 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:09:47.963099 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:09:47.964054 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:47.964549 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:09:47.965044 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:47.965152 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:47.966928 (+ 1776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:09:47.966933 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:09:47.966933 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:09:47.966940 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:09:47.968419 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:09:47.968800 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:09:47.968801 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:09:47.968802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:09:47.968827 (+ 25us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:09:47.968974 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:09:47.968974 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:09:47.968975 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:09:47.969094 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:09:47.969158 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0507 02:09:47.969506 (+ 348us) server_negotiation.cc:299] Negotiation successful
0507 02:09:47.969581 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20240507 02:09:47.969846 14498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:47.962297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:36547 (local address 127.31.131.66:53235)
0507 02:09:47.962435 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:47.962462 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0507 02:09:47.962930 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:09:47.963156 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:09:47.963159 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:09:47.963432 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:09:47.963961 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:47.963966 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:47.964624 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:09:47.964624 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:09:47.964988 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:47.964989 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:47.965082 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:47.965550 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:09:47.965562 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:09:47.966843 (+ 1281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:09:47.968526 (+ 1683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:09:47.968532 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:09:47.968535 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:09:47.968734 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:09:47.968882 (+ 148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:09:47.968883 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:09:47.968883 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:09:47.968931 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:09:47.969180 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:09:47.969189 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:09:47.969357 (+ 168us) client_negotiation.cc:769] Sending connection context
0507 02:09:47.969492 (+ 135us) client_negotiation.cc:240] Negotiation successful
0507 02:09:47.969666 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":65,"threads_started":1}
I20240507 02:09:47.970325 14496 heartbeater.cc:347] Connected to a master server at 127.31.131.126:36547
I20240507 02:09:47.970431 14496 heartbeater.cc:464] Registering TS with master...
I20240507 02:09:47.970664 14496 heartbeater.cc:511] Master 127.31.131.126:36547 requested a full tablet report, sending...
I20240507 02:09:47.971061 14170 ts_manager.cc:194] Registered new tserver with Master: 3b07d0ce58504635ac42242bafc1719a (127.31.131.66:35163)
I20240507 02:09:47.971446 14170 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.131.66@KRBTEST.COM'} at 127.31.131.66:53235
W20240507 02:09:48.045367 14499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:09:48.045545 14499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:09:48.045568 14499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:09:48.046921 14499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:09:48.046964 14499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:09:48.046988 14499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.67
I20240507 02:09:48.048482 14499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:44929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab
--principal=kudu/127.31.131.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.67:43055
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.131.67
--webserver_port=37037
--tserver_master_addrs=127.31.131.126:36547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.67
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:09:48.048686 14499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:09:48.048854 14499 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:09:48.052810 14507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:09:48.052825 14504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:09:48.052981 14505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:09:48.053025 14499 server_base.cc:1034] running on GCE node
I20240507 02:09:48.053277 14499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:09:48.053467 14499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:09:48.054594 14499 hybrid_clock.cc:648] HybridClock initialized: now 1715047788054577 us; error 33 us; skew 500 ppm
May 07 02:09:48 dist-test-slave-v0bg krb5kdc[13593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047788, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:09:48.056883 14499 init.cc:377] Logged in from keytab as kudu/127.31.131.67@KRBTEST.COM (short username kudu)
I20240507 02:09:48.057263 14499 webserver.cc:460] Webserver started at http://127.31.131.67:37037/ using document root <none> and password file <none>
I20240507 02:09:48.057408 14499 fs_manager.cc:359] Metadata directory not provided
I20240507 02:09:48.057452 14499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:09:48.058693 14499 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20240507 02:09:48.059204 14514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:09:48.059378 14499 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240507 02:09:48.059449 14499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-2/wal
uuid: "cb007fe5498b4f9b95aea33b51a36185"
format_stamp: "Formatted at 2024-05-07 02:09:47 on dist-test-slave-v0bg"
server_key: "a8aefd4f61d9cc8dc9f76d6cada64ca6"
server_key_iv: "cd7df357cfd495a12e0f9e36330fd9a1"
server_key_version: "encryptionkey@0"
I20240507 02:09:48.059733 14499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:09:48.078218 14499 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:09:48.080171 14499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:09:48.080281 14499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:09:48.080471 14499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:09:48.080751 14499 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:09:48.080782 14499 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:09:48.080811 14499 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:09:48.080837 14499 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:09:48.085675 14499 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.67:43055
I20240507 02:09:48.085733 14627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.67:43055 every 8 connection(s)
I20240507 02:09:48.086015 14499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
May 07 02:09:48 dist-test-slave-v0bg krb5kdc[13593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047788, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.67@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:09:48.092746 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 14499
I20240507 02:09:48.094080 14361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:48.086647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.67:34505 (local address 127.31.131.126:36547)
0507 02:09:48.086695 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0507 02:09:48.086697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:09:48.086914 (+ 217us) server_negotiation.cc:407] Connection header received
0507 02:09:48.087389 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:09:48.087390 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:09:48.087404 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:09:48.087456 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:09:48.088423 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:48.088922 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:09:48.089445 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:48.089607 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:48.091518 (+ 1911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:09:48.091524 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:09:48.091524 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:09:48.091531 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:09:48.092875 (+ 1344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:09:48.093274 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:09:48.093275 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:09:48.093275 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:09:48.093295 (+ 20us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:09:48.093490 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:09:48.093491 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:09:48.093492 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:09:48.093591 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:09:48.093648 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0507 02:09:48.093897 (+ 249us) server_negotiation.cc:299] Negotiation successful
0507 02:09:48.093987 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20240507 02:09:48.094200 14630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:48.086652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:36547 (local address 127.31.131.67:34505)
0507 02:09:48.086810 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:48.086835 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0507 02:09:48.087293 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:09:48.087511 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:09:48.087515 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:09:48.087794 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:09:48.088320 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:48.088325 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:48.089001 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:09:48.089002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:09:48.089372 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:48.089374 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:48.089497 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:48.090005 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:09:48.090021 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:09:48.091419 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:09:48.092960 (+ 1541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:09:48.092966 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:09:48.092968 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:09:48.093156 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:09:48.093361 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:09:48.093363 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:09:48.093363 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:09:48.093438 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:09:48.093643 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:09:48.093648 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:09:48.093777 (+ 129us) client_negotiation.cc:769] Sending connection context
0507 02:09:48.093870 (+ 93us) client_negotiation.cc:240] Negotiation successful
0507 02:09:48.094003 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":90,"threads_started":1}
I20240507 02:09:48.094633 14628 heartbeater.cc:347] Connected to a master server at 127.31.131.126:36547
I20240507 02:09:48.094729 14628 heartbeater.cc:464] Registering TS with master...
I20240507 02:09:48.094928 14628 heartbeater.cc:511] Master 127.31.131.126:36547 requested a full tablet report, sending...
I20240507 02:09:48.095315 14170 ts_manager.cc:194] Registered new tserver with Master: cb007fe5498b4f9b95aea33b51a36185 (127.31.131.67:43055)
I20240507 02:09:48.095737 14170 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.131.67@KRBTEST.COM'} at 127.31.131.67:34505
I20240507 02:09:48.098374 14632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:48.093107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41860 (local address 127.31.131.126:36547)
0507 02:09:48.093234 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0507 02:09:48.093235 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:09:48.093331 (+ 96us) server_negotiation.cc:407] Connection header received
0507 02:09:48.093393 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:09:48.093394 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:09:48.093406 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:09:48.093456 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:09:48.094098 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:48.094878 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:09:48.095465 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:48.095633 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:48.096522 (+ 889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:09:48.096526 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:09:48.096527 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:09:48.096532 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:09:48.097658 (+ 1126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:09:48.097926 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:09:48.097926 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:09:48.097926 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:09:48.097942 (+ 16us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:09:48.098033 (+ 91us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:09:48.098034 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:09:48.098034 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:09:48.098108 (+ 74us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:09:48.098216 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0507 02:09:48.098247 (+ 31us) server_negotiation.cc:299] Negotiation successful
0507 02:09:48.098307 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":72,"threads_started":1}
I20240507 02:09:48.099054 32269 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240507 02:09:48.101279 14632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:48.099344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41866 (local address 127.31.131.126:36547)
0507 02:09:48.099393 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0507 02:09:48.099395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:09:48.099492 (+ 97us) server_negotiation.cc:407] Connection header received
0507 02:09:48.099561 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:09:48.099561 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:09:48.099583 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:09:48.099641 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0507 02:09:48.099878 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:48.100327 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:09:48.100851 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:48.100975 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:48.101007 (+ 32us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0507 02:09:48.101081 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0507 02:09:48.101143 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0507 02:09:48.101194 (+ 51us) server_negotiation.cc:299] Negotiation successful
0507 02:09:48.101217 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20240507 02:09:48.101524 14170 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:41866:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 02:09:48.103072 14170 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 02:09:48.104833 14292 tablet_service.cc:1467] Processing CreateTablet for tablet a1d1ecdf24174716baceef2af183e91c (DEFAULT_TABLE table=test-table [id=e49ae36b73bc4ee79bf2c0f8855c4138]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:09:48.105022 14292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1d1ecdf24174716baceef2af183e91c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:09:48.106215 14250 tablet_bootstrap.cc:492] T a1d1ecdf24174716baceef2af183e91c P 1bec15577655406baa80dd159df72708: Bootstrap starting.
I20240507 02:09:48.106523 14250 tablet_bootstrap.cc:654] T a1d1ecdf24174716baceef2af183e91c P 1bec15577655406baa80dd159df72708: Neither blocks nor log segments found. Creating new log.
I20240507 02:09:48.107018 14365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:48.104608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.66:35163 (local address 127.0.0.1:48276)
0507 02:09:48.104652 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:48.104662 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0507 02:09:48.104695 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:09:48.105030 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:09:48.105031 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:09:48.105034 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:09:48.105193 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:48.105195 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:48.106128 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:09:48.106129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:09:48.106830 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:48.106833 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:48.106900 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:48.106905 (+ 5us) client_negotiation.cc:769] Sending connection context
0507 02:09:48.106932 (+ 27us) client_negotiation.cc:240] Negotiation successful
0507 02:09:48.106971 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20240507 02:09:48.107120 14250 tablet_bootstrap.cc:492] T a1d1ecdf24174716baceef2af183e91c P 1bec15577655406baa80dd159df72708: No bootstrap required, opened a new log
I20240507 02:09:48.107170 14250 ts_tablet_manager.cc:1367] T a1d1ecdf24174716baceef2af183e91c P 1bec15577655406baa80dd159df72708: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240507 02:09:48.107264 14250 raft_consensus.cc:348] T a1d1ecdf24174716baceef2af183e91c P 1bec15577655406baa80dd159df72708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b07d0ce58504635ac42242bafc1719a" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 35163 } } peers { permanent_uuid: "cb007fe5498b4f9b95aea33b51a36185" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 43055 } } peers { permanent_uuid: "1bec15577655406baa80dd159df72708" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33661 } }
I20240507 02:09:48.107331 14250 raft_consensus.cc:374] T a1d1ecdf24174716baceef2af183e91c P 1bec15577655406baa80dd159df72708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:09:48.107358 14250 raft_consensus.cc:729] T a1d1ecdf24174716baceef2af183e91c P 1bec15577655406baa80dd159df72708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bec15577655406baa80dd159df72708, State: Initialized, Role: FOLLOWER
I20240507 02:09:48.107403 14250 consensus_queue.cc:260] T a1d1ecdf24174716baceef2af183e91c P 1bec15577655406baa80dd159df72708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b07d0ce58504635ac42242bafc1719a" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 35163 } } peers { permanent_uuid: "cb007fe5498b4f9b95aea33b51a36185" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 43055 } } peers { permanent_uuid: "1bec15577655406baa80dd159df72708" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33661 } }
I20240507 02:09:48.107486 14250 ts_tablet_manager.cc:1398] T a1d1ecdf24174716baceef2af183e91c P 1bec15577655406baa80dd159df72708: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240507 02:09:48.107826 14635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:48.104720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48276 (local address 127.31.131.66:35163)
0507 02:09:48.104875 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0507 02:09:48.104884 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0507 02:09:48.104894 (+ 10us) server_negotiation.cc:407] Connection header received
0507 02:09:48.104930 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:09:48.104932 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:09:48.104949 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:09:48.105032 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:09:48.105255 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:48.106061 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:09:48.107136 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:48.107624 (+ 488us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:48.107685 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0507 02:09:48.107718 (+ 33us) server_negotiation.cc:299] Negotiation successful
0507 02:09:48.107755 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":93,"threads_started":1}
I20240507 02:09:48.108327 14430 tablet_service.cc:1467] Processing CreateTablet for tablet a1d1ecdf24174716baceef2af183e91c (DEFAULT_TABLE table=test-table [id=e49ae36b73bc4ee79bf2c0f8855c4138]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:09:48.108608 14430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1d1ecdf24174716baceef2af183e91c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:09:48.109472 14634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:48.104630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.67:43055 (local address 127.0.0.1:40864)
0507 02:09:48.104781 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:48.104789 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0507 02:09:48.104825 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:09:48.106794 (+ 1969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:09:48.106795 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:09:48.106796 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:09:48.106976 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:48.106979 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:48.108442 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:09:48.108443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:09:48.109266 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:48.109269 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:48.109344 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:48.109348 (+ 4us) client_negotiation.cc:769] Sending connection context
0507 02:09:48.109376 (+ 28us) client_negotiation.cc:240] Negotiation successful
0507 02:09:48.109410 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":82,"threads_started":1}
I20240507 02:09:48.109879 14636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:48.106255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40864 (local address 127.31.131.67:43055)
0507 02:09:48.106630 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0507 02:09:48.106637 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0507 02:09:48.106648 (+ 11us) server_negotiation.cc:407] Connection header received
0507 02:09:48.106684 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:09:48.106685 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:09:48.106701 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:09:48.106794 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:09:48.107058 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:48.108363 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:09:48.109350 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:48.109741 (+ 391us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:48.109776 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0507 02:09:48.109802 (+ 26us) server_negotiation.cc:299] Negotiation successful
0507 02:09:48.109828 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":96,"threads_started":1}
I20240507 02:09:48.111182 14638 tablet_bootstrap.cc:492] T a1d1ecdf24174716baceef2af183e91c P 3b07d0ce58504635ac42242bafc1719a: Bootstrap starting.
I20240507 02:09:48.111726 14638 tablet_bootstrap.cc:654] T a1d1ecdf24174716baceef2af183e91c P 3b07d0ce58504635ac42242bafc1719a: Neither blocks nor log segments found. Creating new log.
I20240507 02:09:48.111753 14562 tablet_service.cc:1467] Processing CreateTablet for tablet a1d1ecdf24174716baceef2af183e91c (DEFAULT_TABLE table=test-table [id=e49ae36b73bc4ee79bf2c0f8855c4138]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:09:48.111994 14638 log.cc:824] T a1d1ecdf24174716baceef2af183e91c P 3b07d0ce58504635ac42242bafc1719a: Log is configured to *not* fsync() on all Append() calls
I20240507 02:09:48.112010 14562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1d1ecdf24174716baceef2af183e91c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:09:48.112646 14638 tablet_bootstrap.cc:492] T a1d1ecdf24174716baceef2af183e91c P 3b07d0ce58504635ac42242bafc1719a: No bootstrap required, opened a new log
I20240507 02:09:48.112704 14638 ts_tablet_manager.cc:1367] T a1d1ecdf24174716baceef2af183e91c P 3b07d0ce58504635ac42242bafc1719a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240507 02:09:48.114300 14638 raft_consensus.cc:348] T a1d1ecdf24174716baceef2af183e91c P 3b07d0ce58504635ac42242bafc1719a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b07d0ce58504635ac42242bafc1719a" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 35163 } } peers { permanent_uuid: "cb007fe5498b4f9b95aea33b51a36185" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 43055 } } peers { permanent_uuid: "1bec15577655406baa80dd159df72708" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33661 } }
I20240507 02:09:48.114446 14638 raft_consensus.cc:374] T a1d1ecdf24174716baceef2af183e91c P 3b07d0ce58504635ac42242bafc1719a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:09:48.114473 14638 raft_consensus.cc:729] T a1d1ecdf24174716baceef2af183e91c P 3b07d0ce58504635ac42242bafc1719a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b07d0ce58504635ac42242bafc1719a, State: Initialized, Role: FOLLOWER
I20240507 02:09:48.114610 14638 consensus_queue.cc:260] T a1d1ecdf24174716baceef2af183e91c P 3b07d0ce58504635ac42242bafc1719a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b07d0ce58504635ac42242bafc1719a" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 35163 } } peers { permanent_uuid: "cb007fe5498b4f9b95aea33b51a36185" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 43055 } } peers { permanent_uuid: "1bec15577655406baa80dd159df72708" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33661 } }
I20240507 02:09:48.114849 14638 ts_tablet_manager.cc:1398] T a1d1ecdf24174716baceef2af183e91c P 3b07d0ce58504635ac42242bafc1719a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240507 02:09:48.114835 14640 tablet_bootstrap.cc:492] T a1d1ecdf24174716baceef2af183e91c P cb007fe5498b4f9b95aea33b51a36185: Bootstrap starting.
I20240507 02:09:48.114871 14496 heartbeater.cc:503] Master 127.31.131.126:36547 was elected leader, sending a full tablet report...
I20240507 02:09:48.115371 14640 tablet_bootstrap.cc:654] T a1d1ecdf24174716baceef2af183e91c P cb007fe5498b4f9b95aea33b51a36185: Neither blocks nor log segments found. Creating new log.
I20240507 02:09:48.115641 14640 log.cc:824] T a1d1ecdf24174716baceef2af183e91c P cb007fe5498b4f9b95aea33b51a36185: Log is configured to *not* fsync() on all Append() calls
I20240507 02:09:48.116333 14640 tablet_bootstrap.cc:492] T a1d1ecdf24174716baceef2af183e91c P cb007fe5498b4f9b95aea33b51a36185: No bootstrap required, opened a new log
I20240507 02:09:48.116382 14640 ts_tablet_manager.cc:1367] T a1d1ecdf24174716baceef2af183e91c P cb007fe5498b4f9b95aea33b51a36185: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240507 02:09:48.117887 14640 raft_consensus.cc:348] T a1d1ecdf24174716baceef2af183e91c P cb007fe5498b4f9b95aea33b51a36185 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b07d0ce58504635ac42242bafc1719a" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 35163 } } peers { permanent_uuid: "cb007fe5498b4f9b95aea33b51a36185" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 43055 } } peers { permanent_uuid: "1bec15577655406baa80dd159df72708" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33661 } }
I20240507 02:09:48.117995 14640 raft_consensus.cc:374] T a1d1ecdf24174716baceef2af183e91c P cb007fe5498b4f9b95aea33b51a36185 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:09:48.118014 14640 raft_consensus.cc:729] T a1d1ecdf24174716baceef2af183e91c P cb007fe5498b4f9b95aea33b51a36185 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb007fe5498b4f9b95aea33b51a36185, State: Initialized, Role: FOLLOWER
I20240507 02:09:48.118101 14640 consensus_queue.cc:260] T a1d1ecdf24174716baceef2af183e91c P cb007fe5498b4f9b95aea33b51a36185 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b07d0ce58504635ac42242bafc1719a" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 35163 } } peers { permanent_uuid: "cb007fe5498b4f9b95aea33b51a36185" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 43055 } } peers { permanent_uuid: "1bec15577655406baa80dd159df72708" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33661 } }
I20240507 02:09:48.118321 14640 ts_tablet_manager.cc:1398] T a1d1ecdf24174716baceef2af183e91c P cb007fe5498b4f9b95aea33b51a36185: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240507 02:09:48.118353 14628 heartbeater.cc:503] Master 127.31.131.126:36547 was elected leader, sending a full tablet report...
I20240507 02:09:48.158049 14641 raft_consensus.cc:482] T a1d1ecdf24174716baceef2af183e91c P 3b07d0ce58504635ac42242bafc1719a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 02:09:48.158195 14641 raft_consensus.cc:504] T a1d1ecdf24174716baceef2af183e91c P 3b07d0ce58504635ac42242bafc1719a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b07d0ce58504635ac42242bafc1719a" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 35163 } } peers { permanent_uuid: "cb007fe5498b4f9b95aea33b51a36185" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 43055 } } peers { permanent_uuid: "1bec15577655406baa80dd159df72708" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33661 } }
I20240507 02:09:48.158509 14641 leader_election.cc:288] T a1d1ecdf24174716baceef2af183e91c P 3b07d0ce58504635ac42242bafc1719a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb007fe5498b4f9b95aea33b51a36185 (127.31.131.67:43055), 1bec15577655406baa80dd159df72708 (127.31.131.65:33661)
I20240507 02:09:48.161149 14498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:48.158716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.67:43055 (local address 127.31.131.66:37277)
0507 02:09:48.158794 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:48.158806 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0507 02:09:48.158858 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:09:48.159016 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:09:48.159018 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:09:48.159021 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:09:48.159221 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:48.159223 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:48.160066 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:09:48.160067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:09:48.160899 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:48.160901 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:48.160981 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:48.160986 (+ 5us) client_negotiation.cc:769] Sending connection context
0507 02:09:48.161023 (+ 37us) client_negotiation.cc:240] Negotiation successful
0507 02:09:48.161061 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20240507 02:09:48.161273 14644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:48.158733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.65:33661 (local address 127.31.131.66:57663)
0507 02:09:48.159010 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:48.159017 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0507 02:09:48.159071 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:09:48.159208 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:09:48.159209 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:09:48.159210 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:09:48.159415 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:48.159417 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:48.160214 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:09:48.160215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:09:48.161013 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:48.161015 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:48.161078 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:48.161080 (+ 2us) client_negotiation.cc:769] Sending connection context
0507 02:09:48.161113 (+ 33us) client_negotiation.cc:240] Negotiation successful
0507 02:09:48.161141 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":101,"thread_start_us":119,"threads_started":1}
I20240507 02:09:48.161577 14636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:48.158850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.66:37277 (local address 127.31.131.67:43055)
0507 02:09:48.158897 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0507 02:09:48.158899 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:09:48.158907 (+ 8us) server_negotiation.cc:407] Connection header received
0507 02:09:48.158933 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:09:48.158934 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:09:48.158942 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:09:48.158999 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:09:48.159314 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:48.159984 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:09:48.160991 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:48.161450 (+ 459us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:48.161458 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0507 02:09:48.161479 (+ 21us) server_negotiation.cc:299] Negotiation successful
0507 02:09:48.161509 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20240507 02:09:48.161619 14366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:48.158810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.66:57663 (local address 127.31.131.65:33661)
0507 02:09:48.158864 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0507 02:09:48.158866 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:09:48.159072 (+ 206us) server_negotiation.cc:407] Connection header received
0507 02:09:48.159127 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:09:48.159128 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:09:48.159137 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:09:48.159189 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:09:48.159490 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:48.160134 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:09:48.161090 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:48.161514 (+ 424us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:48.161520 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0507 02:09:48.161538 (+ 18us) server_negotiation.cc:299] Negotiation successful
0507 02:09:48.161564 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20240507 02:09:48.161907 14582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1d1ecdf24174716baceef2af183e91c" candidate_uuid: "3b07d0ce58504635ac42242bafc1719a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb007fe5498b4f9b95aea33b51a36185" is_pre_election: true
I20240507 02:09:48.161906 14312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1d1ecdf24174716baceef2af183e91c" candidate_uuid: "3b07d0ce58504635ac42242bafc1719a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bec15577655406baa80dd159df72708" is_pre_election: true
I20240507 02:09:48.162055 14312 raft_consensus.cc:2454] T a1d1ecdf24174716baceef2af183e91c P 1bec15577655406baa80dd159df72708 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b07d0ce58504635ac42242bafc1719a in term 0.
I20240507 02:09:48.162055 14582 raft_consensus.cc:2454] T a1d1ecdf24174716baceef2af183e91c P cb007fe5498b4f9b95aea33b51a36185 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b07d0ce58504635ac42242bafc1719a in term 0.
I20240507 02:09:48.162274 14384 leader_election.cc:302] T a1d1ecdf24174716baceef2af183e91c P 3b07d0ce58504635ac42242bafc1719a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bec15577655406baa80dd159df72708, 3b07d0ce58504635ac42242bafc1719a; no voters:
I20240507 02:09:48.162405 14641 raft_consensus.cc:2781] T a1d1ecdf24174716baceef2af183e91c P 3b07d0ce58504635ac42242bafc1719a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 02:09:48.162470 14641 raft_consensus.cc:482] T a1d1ecdf24174716baceef2af183e91c P 3b07d0ce58504635ac42242bafc1719a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 02:09:48.162492 14641 raft_consensus.cc:3037] T a1d1ecdf24174716baceef2af183e91c P 3b07d0ce58504635ac42242bafc1719a [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:09:48.163120 14641 raft_consensus.cc:504] T a1d1ecdf24174716baceef2af183e91c P 3b07d0ce58504635ac42242bafc1719a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b07d0ce58504635ac42242bafc1719a" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 35163 } } peers { permanent_uuid: "cb007fe5498b4f9b95aea33b51a36185" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 43055 } } peers { permanent_uuid: "1bec15577655406baa80dd159df72708" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33661 } }
I20240507 02:09:48.163247 14641 leader_election.cc:288] T a1d1ecdf24174716baceef2af183e91c P 3b07d0ce58504635ac42242bafc1719a [CANDIDATE]: Term 1 election: Requested vote from peers cb007fe5498b4f9b95aea33b51a36185 (127.31.131.67:43055), 1bec15577655406baa80dd159df72708 (127.31.131.65:33661)
I20240507 02:09:48.163429 14312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1d1ecdf24174716baceef2af183e91c" candidate_uuid: "3b07d0ce58504635ac42242bafc1719a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bec15577655406baa80dd159df72708"
I20240507 02:09:48.163502 14312 raft_consensus.cc:3037] T a1d1ecdf24174716baceef2af183e91c P 1bec15577655406baa80dd159df72708 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:09:48.164093 14312 raft_consensus.cc:2454] T a1d1ecdf24174716baceef2af183e91c P 1bec15577655406baa80dd159df72708 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b07d0ce58504635ac42242bafc1719a in term 1.
I20240507 02:09:48.164196 14582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1d1ecdf24174716baceef2af183e91c" candidate_uuid: "3b07d0ce58504635ac42242bafc1719a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb007fe5498b4f9b95aea33b51a36185"
I20240507 02:09:48.164256 14384 leader_election.cc:302] T a1d1ecdf24174716baceef2af183e91c P 3b07d0ce58504635ac42242bafc1719a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bec15577655406baa80dd159df72708, 3b07d0ce58504635ac42242bafc1719a; no voters:
I20240507 02:09:48.164275 14582 raft_consensus.cc:3037] T a1d1ecdf24174716baceef2af183e91c P cb007fe5498b4f9b95aea33b51a36185 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:09:48.164327 14641 raft_consensus.cc:2781] T a1d1ecdf24174716baceef2af183e91c P 3b07d0ce58504635ac42242bafc1719a [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:09:48.164479 14641 raft_consensus.cc:686] T a1d1ecdf24174716baceef2af183e91c P 3b07d0ce58504635ac42242bafc1719a [term 1 LEADER]: Becoming Leader. State: Replica: 3b07d0ce58504635ac42242bafc1719a, State: Running, Role: LEADER
I20240507 02:09:48.164651 14641 consensus_queue.cc:237] T a1d1ecdf24174716baceef2af183e91c P 3b07d0ce58504635ac42242bafc1719a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b07d0ce58504635ac42242bafc1719a" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 35163 } } peers { permanent_uuid: "cb007fe5498b4f9b95aea33b51a36185" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 43055 } } peers { permanent_uuid: "1bec15577655406baa80dd159df72708" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33661 } }
I20240507 02:09:48.165100 14582 raft_consensus.cc:2454] T a1d1ecdf24174716baceef2af183e91c P cb007fe5498b4f9b95aea33b51a36185 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b07d0ce58504635ac42242bafc1719a in term 1.
I20240507 02:09:48.165447 14170 catalog_manager.cc:5484] T a1d1ecdf24174716baceef2af183e91c P 3b07d0ce58504635ac42242bafc1719a reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b07d0ce58504635ac42242bafc1719a (127.31.131.66). New cstate: current_term: 1 leader_uuid: "3b07d0ce58504635ac42242bafc1719a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b07d0ce58504635ac42242bafc1719a" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 35163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb007fe5498b4f9b95aea33b51a36185" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bec15577655406baa80dd159df72708" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33661 } health_report { overall_health: UNKNOWN } } }
I20240507 02:09:48.182430 14635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:48.180532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48286 (local address 127.31.131.66:35163)
0507 02:09:48.180592 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0507 02:09:48.180594 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:09:48.180602 (+ 8us) server_negotiation.cc:407] Connection header received
0507 02:09:48.180639 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:09:48.180640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:09:48.180656 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:09:48.180721 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0507 02:09:48.180934 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:48.181420 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:09:48.182005 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:48.182141 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:48.182177 (+ 36us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0507 02:09:48.182264 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0507 02:09:48.182329 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0507 02:09:48.182345 (+ 16us) server_negotiation.cc:299] Negotiation successful
0507 02:09:48.182363 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20240507 02:09:48.183948 14312 raft_consensus.cc:1261] T a1d1ecdf24174716baceef2af183e91c P 1bec15577655406baa80dd159df72708 [term 1 FOLLOWER]: Refusing update from remote peer 3b07d0ce58504635ac42242bafc1719a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 02:09:48.183946 14582 raft_consensus.cc:1261] T a1d1ecdf24174716baceef2af183e91c P cb007fe5498b4f9b95aea33b51a36185 [term 1 FOLLOWER]: Refusing update from remote peer 3b07d0ce58504635ac42242bafc1719a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 02:09:48.184248 14641 consensus_queue.cc:1035] T a1d1ecdf24174716baceef2af183e91c P 3b07d0ce58504635ac42242bafc1719a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb007fe5498b4f9b95aea33b51a36185" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 02:09:48.184372 14645 consensus_queue.cc:1035] T a1d1ecdf24174716baceef2af183e91c P 3b07d0ce58504635ac42242bafc1719a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bec15577655406baa80dd159df72708" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 02:09:48.186486 14649 mvcc.cc:205] Tried to move back new op lower bound from 7024835740399329280 to 7024835740323123200. Current Snapshot: MvccSnapshot[applied={T|T < 7024835740399329280}]
I20240507 02:09:48.187757 14650 mvcc.cc:205] Tried to move back new op lower bound from 7024835740399329280 to 7024835740323123200. Current Snapshot: MvccSnapshot[applied={T|T < 7024835740399329280 or (T in {7024835740399329280})}]
I20240507 02:09:48.188351 14169 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:41866:
table { table_name: "test-table" } modify_external_catalogs: true
I20240507 02:09:48.188438 14169 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:41866:
table { table_name: "test-table" } modify_external_catalogs: true
I20240507 02:09:48.189074 14169 catalog_manager.cc:5771] T 00000000000000000000000000000000 P b3f5a6f84b18408c94e6a9c6e49ce384: Sending DeleteTablet for 3 replicas of tablet a1d1ecdf24174716baceef2af183e91c
I20240507 02:09:48.189358 14562 tablet_service.cc:1514] Processing DeleteTablet for tablet a1d1ecdf24174716baceef2af183e91c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:09:48 UTC) from {username='kudu', principal='kudu/127.31.131.126@KRBTEST.COM'} at 127.0.0.1:40864
I20240507 02:09:48.189428 14292 tablet_service.cc:1514] Processing DeleteTablet for tablet a1d1ecdf24174716baceef2af183e91c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:09:48 UTC) from {username='kudu', principal='kudu/127.31.131.126@KRBTEST.COM'} at 127.0.0.1:60144
I20240507 02:09:48.189520 14367 tablet_replica.cc:330] T a1d1ecdf24174716baceef2af183e91c P 1bec15577655406baa80dd159df72708: stopping tablet replica
I20240507 02:09:48.189576 14367 raft_consensus.cc:2229] T a1d1ecdf24174716baceef2af183e91c P 1bec15577655406baa80dd159df72708 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 02:09:48.189591 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 14208
I20240507 02:09:48.189637 14367 raft_consensus.cc:2258] T a1d1ecdf24174716baceef2af183e91c P 1bec15577655406baa80dd159df72708 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 02:09:48.189633 14430 tablet_service.cc:1514] Processing DeleteTablet for tablet a1d1ecdf24174716baceef2af183e91c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:09:48 UTC) from {username='kudu', principal='kudu/127.31.131.126@KRBTEST.COM'} at 127.0.0.1:48276
I20240507 02:09:48.190297 14658 tablet_replica.cc:330] T a1d1ecdf24174716baceef2af183e91c P cb007fe5498b4f9b95aea33b51a36185: stopping tablet replica
I20240507 02:09:48.190353 14658 raft_consensus.cc:2229] T a1d1ecdf24174716baceef2af183e91c P cb007fe5498b4f9b95aea33b51a36185 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 02:09:48.190428 14658 raft_consensus.cc:2258] T a1d1ecdf24174716baceef2af183e91c P cb007fe5498b4f9b95aea33b51a36185 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 02:09:48.190973 14658 ts_tablet_manager.cc:1875] T a1d1ecdf24174716baceef2af183e91c P cb007fe5498b4f9b95aea33b51a36185: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 02:09:48.192459 14658 ts_tablet_manager.cc:1888] T a1d1ecdf24174716baceef2af183e91c P cb007fe5498b4f9b95aea33b51a36185: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240507 02:09:48.192523 14658 log.cc:1196] T a1d1ecdf24174716baceef2af183e91c P cb007fe5498b4f9b95aea33b51a36185: Deleting WAL directory at /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-2/wal/wals/a1d1ecdf24174716baceef2af183e91c
I20240507 02:09:48.192689 14659 tablet_replica.cc:330] T a1d1ecdf24174716baceef2af183e91c P 3b07d0ce58504635ac42242bafc1719a: stopping tablet replica
I20240507 02:09:48.192762 14658 ts_tablet_manager.cc:1909] T a1d1ecdf24174716baceef2af183e91c P cb007fe5498b4f9b95aea33b51a36185: Deleting consensus metadata
I20240507 02:09:48.192865 14659 raft_consensus.cc:2229] T a1d1ecdf24174716baceef2af183e91c P 3b07d0ce58504635ac42242bafc1719a [term 1 LEADER]: Raft consensus shutting down.
I20240507 02:09:48.192984 14659 raft_consensus.cc:2258] T a1d1ecdf24174716baceef2af183e91c P 3b07d0ce58504635ac42242bafc1719a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 02:09:48.193428 14659 ts_tablet_manager.cc:1875] T a1d1ecdf24174716baceef2af183e91c P 3b07d0ce58504635ac42242bafc1719a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240507 02:09:48.194372 14155 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.131.65:33661 (error 108)
I20240507 02:09:48.194658 14154 catalog_manager.cc:4830] TS cb007fe5498b4f9b95aea33b51a36185 (127.31.131.67:43055): tablet a1d1ecdf24174716baceef2af183e91c (table test-table [id=e49ae36b73bc4ee79bf2c0f8855c4138]) successfully deleted
I20240507 02:09:48.194974 14659 ts_tablet_manager.cc:1888] T a1d1ecdf24174716baceef2af183e91c P 3b07d0ce58504635ac42242bafc1719a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240507 02:09:48.195044 14659 log.cc:1196] T a1d1ecdf24174716baceef2af183e91c P 3b07d0ce58504635ac42242bafc1719a: Deleting WAL directory at /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-1/wal/wals/a1d1ecdf24174716baceef2af183e91c
W20240507 02:09:48.195237 14634 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:09:48.195099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.65:33661 (local address 127.0.0.1:60150)
0507 02:09:48.195149 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:48.195160 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0507 02:09:48.195191 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.65:33661: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":31}
I20240507 02:09:48.195299 14659 ts_tablet_manager.cc:1909] T a1d1ecdf24174716baceef2af183e91c P 3b07d0ce58504635ac42242bafc1719a: Deleting consensus metadata
W20240507 02:09:48.195331 14155 catalog_manager.cc:4573] TS 1bec15577655406baa80dd159df72708 (127.31.131.65:33661): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a1d1ecdf24174716baceef2af183e91c: Network error: Client connection negotiation failed: client connection to 127.31.131.65:33661: BlockingWrite error: write error: Connection reset by peer (error 104)
I20240507 02:09:48.195633 14157 catalog_manager.cc:4830] TS 3b07d0ce58504635ac42242bafc1719a (127.31.131.66:35163): tablet a1d1ecdf24174716baceef2af183e91c (table test-table [id=e49ae36b73bc4ee79bf2c0f8855c4138]) successfully deleted
I20240507 02:09:48.195878 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 14364
I20240507 02:09:48.200325 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 14499
I20240507 02:09:48.205806 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 14137
I20240507 02:09:48.210041 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:36547
--webserver_interface=127.31.131.126
--webserver_port=40679
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab
--principal=kudu/127.31.131.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:44929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.131.126:36547
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:09:48.286473 14661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:09:48.286767 14661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:09:48.286811 14661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:09:48.288151 14661 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240507 02:09:48.288194 14661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:09:48.288208 14661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:09:48.288221 14661 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240507 02:09:48.288244 14661 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240507 02:09:48.289755 14661 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:44929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.131.126:36547
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab
--principal=kudu/127.31.131.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:36547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.131.126
--webserver_port=40679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14661
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:09:48.289954 14661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:09:48.290128 14661 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:09:48.293807 14666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:09:48.293807 14669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:09:48.293819 14661 server_base.cc:1034] running on GCE node
W20240507 02:09:48.293995 14667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:09:48.294217 14661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:09:48.294435 14661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:09:48.295557 14661 hybrid_clock.cc:648] HybridClock initialized: now 1715047788295539 us; error 31 us; skew 500 ppm
May 07 02:09:48 dist-test-slave-v0bg krb5kdc[13593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047788, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:09:48.297698 14661 init.cc:377] Logged in from keytab as kudu/127.31.131.126@KRBTEST.COM (short username kudu)
I20240507 02:09:48.298074 14661 webserver.cc:460] Webserver started at http://127.31.131.126:40679/ using document root <none> and password file <none>
I20240507 02:09:48.298229 14661 fs_manager.cc:359] Metadata directory not provided
I20240507 02:09:48.298264 14661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:09:48.299398 14661 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20240507 02:09:48.299897 14676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:09:48.300043 14661 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240507 02:09:48.300108 14661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/master-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/master-0/wal
uuid: "b3f5a6f84b18408c94e6a9c6e49ce384"
format_stamp: "Formatted at 2024-05-07 02:09:47 on dist-test-slave-v0bg"
server_key: "784f36bbe8cea968188f2cc04d2c7033"
server_key_iv: "8af29a7f465f9744faff075c642b1967"
server_key_version: "encryptionkey@0"
I20240507 02:09:48.300374 14661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:09:48.323370 14661 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:09:48.324965 14661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:09:48.325062 14661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:09:48.328361 14661 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.126:36547
I20240507 02:09:48.328408 14728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.126:36547 every 8 connection(s)
I20240507 02:09:48.328699 14661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
I20240507 02:09:48.329962 14729 sys_catalog.cc:263] Verifying existing consensus state
I20240507 02:09:48.330451 14729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3f5a6f84b18408c94e6a9c6e49ce384: Bootstrap starting.
I20240507 02:09:48.333781 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 14661
I20240507 02:09:48.334014 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.65:33661
--local_ip_for_outbound_sockets=127.31.131.65
--tserver_master_addrs=127.31.131.126:36547
--webserver_port=41741
--webserver_interface=127.31.131.65
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab
--principal=kudu/127.31.131.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:44929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240507 02:09:48.339144 14729 log.cc:824] T 00000000000000000000000000000000 P b3f5a6f84b18408c94e6a9c6e49ce384: Log is configured to *not* fsync() on all Append() calls
I20240507 02:09:48.341004 14729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3f5a6f84b18408c94e6a9c6e49ce384: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240507 02:09:48.341219 14729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3f5a6f84b18408c94e6a9c6e49ce384: Bootstrap complete.
I20240507 02:09:48.343313 14729 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3f5a6f84b18408c94e6a9c6e49ce384 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3f5a6f84b18408c94e6a9c6e49ce384" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 36547 } }
I20240507 02:09:48.343506 14729 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3f5a6f84b18408c94e6a9c6e49ce384 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3f5a6f84b18408c94e6a9c6e49ce384, State: Initialized, Role: FOLLOWER
I20240507 02:09:48.343631 14729 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3f5a6f84b18408c94e6a9c6e49ce384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3f5a6f84b18408c94e6a9c6e49ce384" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 36547 } }
I20240507 02:09:48.343698 14729 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3f5a6f84b18408c94e6a9c6e49ce384 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 02:09:48.343730 14729 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3f5a6f84b18408c94e6a9c6e49ce384 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 02:09:48.343773 14729 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3f5a6f84b18408c94e6a9c6e49ce384 [term 2 FOLLOWER]: Advancing to term 3
I20240507 02:09:48.344599 14729 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3f5a6f84b18408c94e6a9c6e49ce384 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3f5a6f84b18408c94e6a9c6e49ce384" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 36547 } }
I20240507 02:09:48.344691 14729 leader_election.cc:302] T 00000000000000000000000000000000 P b3f5a6f84b18408c94e6a9c6e49ce384 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3f5a6f84b18408c94e6a9c6e49ce384; no voters:
I20240507 02:09:48.344826 14729 leader_election.cc:288] T 00000000000000000000000000000000 P b3f5a6f84b18408c94e6a9c6e49ce384 [CANDIDATE]: Term 3 election: Requested vote from peers
I20240507 02:09:48.344889 14733 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3f5a6f84b18408c94e6a9c6e49ce384 [term 3 FOLLOWER]: Leader election won for term 3
I20240507 02:09:48.345037 14729 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3f5a6f84b18408c94e6a9c6e49ce384 [sys.catalog]: configured and running, proceeding with master startup.
I20240507 02:09:48.345089 14733 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3f5a6f84b18408c94e6a9c6e49ce384 [term 3 LEADER]: Becoming Leader. State: Replica: b3f5a6f84b18408c94e6a9c6e49ce384, State: Running, Role: LEADER
I20240507 02:09:48.345165 14733 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3f5a6f84b18408c94e6a9c6e49ce384 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3f5a6f84b18408c94e6a9c6e49ce384" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 36547 } }
I20240507 02:09:48.345413 14733 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3f5a6f84b18408c94e6a9c6e49ce384 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3f5a6f84b18408c94e6a9c6e49ce384. Latest consensus state: current_term: 3 leader_uuid: "b3f5a6f84b18408c94e6a9c6e49ce384" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3f5a6f84b18408c94e6a9c6e49ce384" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 36547 } } }
I20240507 02:09:48.345472 14733 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3f5a6f84b18408c94e6a9c6e49ce384 [sys.catalog]: This master's current role is: LEADER
I20240507 02:09:48.345585 14734 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3f5a6f84b18408c94e6a9c6e49ce384 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "b3f5a6f84b18408c94e6a9c6e49ce384" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3f5a6f84b18408c94e6a9c6e49ce384" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 36547 } } }
I20240507 02:09:48.345629 14734 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3f5a6f84b18408c94e6a9c6e49ce384 [sys.catalog]: This master's current role is: LEADER
I20240507 02:09:48.345702 14739 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 02:09:48.346357 14739 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240507 02:09:48.346450 14739 catalog_manager.cc:1259] Loaded cluster ID: a212f29e4e124c83a118b6754b8cae85
I20240507 02:09:48.346477 14739 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20240507 02:09:48.346669 14739 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:asn1_check_tlen:wrong tag:../crypto/asn1/tasn_dec.c:1149 error:0D06C03A:asn1 encoding routines:asn1_d2i_ex_primitive:nested asn1 error:../crypto/asn1/tasn_dec.c:713 error:0D08303A:asn1 encoding routines:asn1_template_noexp_d2i:nested asn1 error:../crypto/asn1/tasn_dec.c:646:Field=version, Type=RSAPrivateKey error:04093004:rsa routines:old_rsa_priv_decode:RSA lib:../crypto/rsa/rsa_ameth.c:130 error:0D0680A8:asn1 encoding routines:asn1_check_tlen:wrong tag:../crypto/asn1/tasn_dec.c:1149 error:0D06C03A:asn1 encoding routines:asn1_d2i_ex_primitive:nested asn1 error:../crypto/asn1/tasn_dec.c:713 error:0D08303A:asn1 encoding routines:asn1_template_noexp_d2i:nested asn1 error:../crypto/asn1/tasn_dec.c:646:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f028615cdcd google::LogMessage::Fail() at ??:0
@ 0x7f0286160b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f028615c7cc google::LogMessage::Flush() at ??:0
@ 0x7f028615df59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f0288935af6 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f0288940294 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f0287256a08 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f028724b3a4 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f028756e6db start_thread at ??:0
@ 0x7f02856b171f clone at ??:0
W20240507 02:09:48.421111 14731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:09:48.421288 14731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:09:48.421308 14731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:09:48.422608 14731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:09:48.422647 14731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:09:48.422670 14731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.65
I20240507 02:09:48.424074 14731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:44929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab
--principal=kudu/127.31.131.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.65:33661
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.131.65
--webserver_port=41741
--tserver_master_addrs=127.31.131.126:36547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.65
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:09:48.424255 14731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:09:48.424422 14731 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:09:48.428023 14768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:09:48.428030 14765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:09:48.428264 14731 server_base.cc:1034] running on GCE node
W20240507 02:09:48.428282 14766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:09:48.428468 14731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:09:48.428674 14731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:09:48.429795 14731 hybrid_clock.cc:648] HybridClock initialized: now 1715047788429785 us; error 30 us; skew 500 ppm
May 07 02:09:48 dist-test-slave-v0bg krb5kdc[13593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047788, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:09:48.435308 14731 init.cc:377] Logged in from keytab as kudu/127.31.131.65@KRBTEST.COM (short username kudu)
I20240507 02:09:48.435695 14731 webserver.cc:460] Webserver started at http://127.31.131.65:41741/ using document root <none> and password file <none>
I20240507 02:09:48.435850 14731 fs_manager.cc:359] Metadata directory not provided
I20240507 02:09:48.435891 14731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:09:48.437049 14731 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20240507 02:09:48.437613 14775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:09:48.437783 14731 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240507 02:09:48.437852 14731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-0/wal
uuid: "1bec15577655406baa80dd159df72708"
format_stamp: "Formatted at 2024-05-07 02:09:47 on dist-test-slave-v0bg"
server_key: "8632f4ad1e8414ee6fb17ee667ac2f6e"
server_key_iv: "ec4f18d4430225476621a09832fcd7ea"
server_key_version: "encryptionkey@0"
I20240507 02:09:48.438127 14731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:09:48.449539 14731 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:09:48.451380 14731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:09:48.451479 14731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:09:48.451658 14731 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:09:48.452014 14782 ts_tablet_manager.cc:516] Loading tablet metadata (0/1 complete)
I20240507 02:09:48.453032 14731 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240507 02:09:48.453069 14731 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20240507 02:09:48.453100 14731 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240507 02:09:48.453613 14731 ts_tablet_manager.cc:590] Registered 1 tablets
I20240507 02:09:48.453644 14731 ts_tablet_manager.cc:569] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20240507 02:09:48.453720 14782 tablet_bootstrap.cc:492] T a1d1ecdf24174716baceef2af183e91c P 1bec15577655406baa80dd159df72708: Bootstrap starting.
I20240507 02:09:48.460042 14731 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.65:33661
I20240507 02:09:48.460362 14731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
I20240507 02:09:48.460837 14889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.65:33661 every 8 connection(s)
I20240507 02:09:48.465624 14782 log.cc:824] T a1d1ecdf24174716baceef2af183e91c P 1bec15577655406baa80dd159df72708: Log is configured to *not* fsync() on all Append() calls
I20240507 02:09:48.468101 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 14731
I20240507 02:09:48.468130 14782 tablet_bootstrap.cc:492] T a1d1ecdf24174716baceef2af183e91c P 1bec15577655406baa80dd159df72708: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240507 02:09:48.468350 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.66:35163
--local_ip_for_outbound_sockets=127.31.131.66
--tserver_master_addrs=127.31.131.126:36547
--webserver_port=37415
--webserver_interface=127.31.131.66
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab
--principal=kudu/127.31.131.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:44929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240507 02:09:48.468413 14782 tablet_bootstrap.cc:492] T a1d1ecdf24174716baceef2af183e91c P 1bec15577655406baa80dd159df72708: Bootstrap complete.
I20240507 02:09:48.469491 14782 ts_tablet_manager.cc:1367] T a1d1ecdf24174716baceef2af183e91c P 1bec15577655406baa80dd159df72708: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.009s
I20240507 02:09:48.470743 14782 raft_consensus.cc:348] T a1d1ecdf24174716baceef2af183e91c P 1bec15577655406baa80dd159df72708 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b07d0ce58504635ac42242bafc1719a" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 35163 } } peers { permanent_uuid: "cb007fe5498b4f9b95aea33b51a36185" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 43055 } } peers { permanent_uuid: "1bec15577655406baa80dd159df72708" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33661 } }
I20240507 02:09:48.471047 14782 raft_consensus.cc:729] T a1d1ecdf24174716baceef2af183e91c P 1bec15577655406baa80dd159df72708 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bec15577655406baa80dd159df72708, State: Initialized, Role: FOLLOWER
I20240507 02:09:48.471204 14782 consensus_queue.cc:260] T a1d1ecdf24174716baceef2af183e91c P 1bec15577655406baa80dd159df72708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b07d0ce58504635ac42242bafc1719a" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 35163 } } peers { permanent_uuid: "cb007fe5498b4f9b95aea33b51a36185" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 43055 } } peers { permanent_uuid: "1bec15577655406baa80dd159df72708" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33661 } }
I20240507 02:09:48.471482 14782 ts_tablet_manager.cc:1398] T a1d1ecdf24174716baceef2af183e91c P 1bec15577655406baa80dd159df72708: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20240507 02:09:48.525501 14892 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:09:48.460989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:36547 (local address 127.31.131.65:59871)
0507 02:09:48.461137 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:48.461166 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0507 02:09:48.461588 (+ 422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:09:48.525328 (+ 63740us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:36547: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":88,"threads_started":1}
W20240507 02:09:48.525787 14890 heartbeater.cc:644] Failed to heartbeat to 127.31.131.126:36547 (0 consecutive failures): Network error: Failed to ping master at 127.31.131.126:36547: Client connection negotiation failed: client connection to 127.31.131.126:36547: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240507 02:09:48.526135 14892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:48.526020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:36547 (local address 127.31.131.65:44547)
0507 02:09:48.526068 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:48.526110 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:36547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20240507 02:09:48.526456 14892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:48.526381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:36547 (local address 127.31.131.65:32813)
0507 02:09:48.526416 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:48.526439 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:36547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20240507 02:09:48.526544 14890 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20240507 02:09:48.547009 14894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:09:48.547192 14894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:09:48.547214 14894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:09:48.548432 14894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:09:48.548467 14894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:09:48.548491 14894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.66
I20240507 02:09:48.549891 14894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:44929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab
--principal=kudu/127.31.131.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.66:35163
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.131.66
--webserver_port=37415
--tserver_master_addrs=127.31.131.126:36547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.66
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:09:48.550050 14894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:09:48.550202 14894 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:09:48.553848 14901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:09:48.553843 14903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:09:48.553927 14894 server_base.cc:1034] running on GCE node
W20240507 02:09:48.553843 14900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:09:48.554198 14894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:09:48.554373 14894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:09:48.555506 14894 hybrid_clock.cc:648] HybridClock initialized: now 1715047788555496 us; error 29 us; skew 500 ppm
May 07 02:09:48 dist-test-slave-v0bg krb5kdc[13593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047788, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:09:48.557745 14894 init.cc:377] Logged in from keytab as kudu/127.31.131.66@KRBTEST.COM (short username kudu)
I20240507 02:09:48.558110 14894 webserver.cc:460] Webserver started at http://127.31.131.66:37415/ using document root <none> and password file <none>
I20240507 02:09:48.558282 14894 fs_manager.cc:359] Metadata directory not provided
I20240507 02:09:48.558317 14894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:09:48.559439 14894 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20240507 02:09:48.559896 14910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:09:48.560050 14894 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240507 02:09:48.560117 14894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-1/wal
uuid: "3b07d0ce58504635ac42242bafc1719a"
format_stamp: "Formatted at 2024-05-07 02:09:47 on dist-test-slave-v0bg"
server_key: "3c13638d8c8706f0a6d9f32a285502e2"
server_key_iv: "57b4055fe2a88d95fd23cda7ed4ea520"
server_key_version: "encryptionkey@0"
I20240507 02:09:48.560384 14894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:09:48.572882 14894 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:09:48.574350 14894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:09:48.574447 14894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:09:48.574651 14894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:09:48.574914 14894 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:09:48.574941 14894 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:09:48.574965 14894 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:09:48.574980 14894 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:09:48.580176 14894 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.66:35163
I20240507 02:09:48.580238 15023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.66:35163 every 8 connection(s)
I20240507 02:09:48.580510 14894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
I20240507 02:09:48.581351 15026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:48.581058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:36547 (local address 127.31.131.66:55875)
0507 02:09:48.581171 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:48.581242 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:36547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":55,"threads_started":1}
W20240507 02:09:48.581492 15024 heartbeater.cc:644] Failed to heartbeat to 127.31.131.126:36547 (0 consecutive failures): Network error: Failed to ping master at 127.31.131.126:36547: Client connection negotiation failed: client connection to 127.31.131.126:36547: connect: Connection refused (error 111)
I20240507 02:09:48.581708 15026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:48.581636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:36547 (local address 127.31.131.66:60459)
0507 02:09:48.581672 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:48.581693 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:36547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20240507 02:09:48.581895 15026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:48.581840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:36547 (local address 127.31.131.66:34203)
0507 02:09:48.581869 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:48.581885 (+ 16us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:36547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20240507 02:09:48.581959 15024 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20240507 02:09:48.583187 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 14894
I20240507 02:09:48.583425 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.67:43055
--local_ip_for_outbound_sockets=127.31.131.67
--tserver_master_addrs=127.31.131.126:36547
--webserver_port=37037
--webserver_interface=127.31.131.67
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab
--principal=kudu/127.31.131.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:44929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:09:48.658447 15027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:09:48.658658 15027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:09:48.658699 15027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:09:48.659933 15027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:09:48.659972 15027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:09:48.660004 15027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.67
I20240507 02:09:48.661430 15027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:44929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab
--principal=kudu/127.31.131.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.67:43055
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.131.67
--webserver_port=37037
--tserver_master_addrs=127.31.131.126:36547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.67
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:09:48.661634 15027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:09:48.661826 15027 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:09:48.665591 15033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:09:48.665622 15032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:09:48.665747 15027 server_base.cc:1034] running on GCE node
W20240507 02:09:48.665745 15035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:09:48.665932 15027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:09:48.666097 15027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:09:48.667222 15027 hybrid_clock.cc:648] HybridClock initialized: now 1715047788667205 us; error 32 us; skew 500 ppm
May 07 02:09:48 dist-test-slave-v0bg krb5kdc[13593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047788, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:09:48.669313 15027 init.cc:377] Logged in from keytab as kudu/127.31.131.67@KRBTEST.COM (short username kudu)
I20240507 02:09:48.669652 15027 webserver.cc:460] Webserver started at http://127.31.131.67:37037/ using document root <none> and password file <none>
I20240507 02:09:48.669806 15027 fs_manager.cc:359] Metadata directory not provided
I20240507 02:09:48.669842 15027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:09:48.670962 15027 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20240507 02:09:48.671407 15042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:09:48.671548 15027 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240507 02:09:48.671617 15027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-2/wal
uuid: "cb007fe5498b4f9b95aea33b51a36185"
format_stamp: "Formatted at 2024-05-07 02:09:47 on dist-test-slave-v0bg"
server_key: "a8aefd4f61d9cc8dc9f76d6cada64ca6"
server_key_iv: "cd7df357cfd495a12e0f9e36330fd9a1"
server_key_version: "encryptionkey@0"
I20240507 02:09:48.671867 15027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:09:48.685453 15027 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:09:48.686899 15027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:09:48.687003 15027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:09:48.687170 15027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:09:48.687441 15027 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:09:48.687471 15027 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:09:48.687497 15027 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:09:48.687525 15027 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:09:48.692414 15027 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.67:43055
I20240507 02:09:48.692451 15155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.67:43055 every 8 connection(s)
I20240507 02:09:48.692719 15027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
I20240507 02:09:48.693531 15158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:48.693256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:36547 (local address 127.31.131.67:48459)
0507 02:09:48.693373 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:48.693427 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:36547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":57,"threads_started":1}
W20240507 02:09:48.693701 15156 heartbeater.cc:644] Failed to heartbeat to 127.31.131.126:36547 (0 consecutive failures): Network error: Failed to ping master at 127.31.131.126:36547: Client connection negotiation failed: client connection to 127.31.131.126:36547: connect: Connection refused (error 111)
I20240507 02:09:48.693913 15158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:48.693845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:36547 (local address 127.31.131.67:43025)
0507 02:09:48.693878 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:48.693899 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:36547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20240507 02:09:48.694098 15158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:48.694047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:36547 (local address 127.31.131.67:38719)
0507 02:09:48.694073 (+ 26us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:48.694088 (+ 15us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:36547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20240507 02:09:48.694169 15156 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20240507 02:09:48.696988 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 15027
I20240507 02:09:49.527135 15160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:49.526915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:36547 (local address 127.31.131.65:59041)
0507 02:09:49.527036 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:49.527086 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:36547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":64,"threads_started":1}
I20240507 02:09:49.582487 15161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:49.582260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:36547 (local address 127.31.131.66:41531)
0507 02:09:49.582395 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:49.582430 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:36547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":82,"threads_started":1}
I20240507 02:09:49.694736 15162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:49.694486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:36547 (local address 127.31.131.67:54809)
0507 02:09:49.694632 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:49.694671 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:36547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":95,"threads_started":1}
I20240507 02:09:50.241261 15163 raft_consensus.cc:482] T a1d1ecdf24174716baceef2af183e91c P 1bec15577655406baa80dd159df72708 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 02:09:50.241389 15163 raft_consensus.cc:504] T a1d1ecdf24174716baceef2af183e91c P 1bec15577655406baa80dd159df72708 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b07d0ce58504635ac42242bafc1719a" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 35163 } } peers { permanent_uuid: "cb007fe5498b4f9b95aea33b51a36185" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 43055 } } peers { permanent_uuid: "1bec15577655406baa80dd159df72708" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33661 } }
I20240507 02:09:50.241690 15163 leader_election.cc:288] T a1d1ecdf24174716baceef2af183e91c P 1bec15577655406baa80dd159df72708 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3b07d0ce58504635ac42242bafc1719a (127.31.131.66:35163), cb007fe5498b4f9b95aea33b51a36185 (127.31.131.67:43055)
May 07 02:09:50 dist-test-slave-v0bg krb5kdc[13593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047788, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.65@KRBTEST.COM for kudu/127.31.131.67@KRBTEST.COM
May 07 02:09:50 dist-test-slave-v0bg krb5kdc[13593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047788, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.65@KRBTEST.COM for kudu/127.31.131.66@KRBTEST.COM
I20240507 02:09:50.250063 15166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:50.241920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:40399 (local address 127.31.131.66:35163)
0507 02:09:50.242085 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0507 02:09:50.242095 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0507 02:09:50.242116 (+ 21us) server_negotiation.cc:407] Connection header received
0507 02:09:50.242530 (+ 414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:09:50.242532 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:09:50.242599 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:09:50.242726 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:09:50.243530 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:50.244225 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:09:50.244718 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:50.244934 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:50.247224 (+ 2290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:09:50.247231 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:09:50.247232 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:09:50.247246 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:09:50.248769 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:09:50.249151 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:09:50.249154 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:09:50.249154 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:09:50.249204 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:09:50.249373 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:09:50.249374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:09:50.249375 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:09:50.249554 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:09:50.249627 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0507 02:09:50.249862 (+ 235us) server_negotiation.cc:299] Negotiation successful
0507 02:09:50.249955 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":74,"threads_started":1}
I20240507 02:09:50.250069 15167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:50.242036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:43607 (local address 127.31.131.67:43055)
0507 02:09:50.242153 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0507 02:09:50.242160 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0507 02:09:50.242173 (+ 13us) server_negotiation.cc:407] Connection header received
0507 02:09:50.242533 (+ 360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:09:50.242535 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:09:50.242605 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:09:50.242725 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:09:50.243530 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:50.244225 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:09:50.244725 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:09:50.244935 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:50.246906 (+ 1971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:09:50.246916 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:09:50.246916 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:09:50.246937 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:09:50.248727 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:09:50.249158 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:09:50.249160 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:09:50.249161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:09:50.249212 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:09:50.249373 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:09:50.249374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:09:50.249374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:09:50.249555 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:09:50.249627 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0507 02:09:50.249866 (+ 239us) server_negotiation.cc:299] Negotiation successful
0507 02:09:50.249968 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":58,"threads_started":1}
I20240507 02:09:50.250105 15164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:50.241837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.66:35163 (local address 127.31.131.65:40399)
0507 02:09:50.242016 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:50.242026 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0507 02:09:50.242089 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:09:50.242763 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:09:50.242766 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:09:50.243180 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:09:50.243428 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:50.243433 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:50.244318 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:09:50.244319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:09:50.244652 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:50.244653 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:50.244765 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:50.245434 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:09:50.245437 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:09:50.247140 (+ 1703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:09:50.248843 (+ 1703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:09:50.248844 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:09:50.248844 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:09:50.249084 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:09:50.249260 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:09:50.249261 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:09:50.249261 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:09:50.249320 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:09:50.249615 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:09:50.249620 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:09:50.249717 (+ 97us) client_negotiation.cc:769] Sending connection context
0507 02:09:50.249805 (+ 88us) client_negotiation.cc:240] Negotiation successful
0507 02:09:50.249947 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":11,"thread_start_us":33,"threads_started":1}
I20240507 02:09:50.250229 15165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:50.241837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.67:43055 (local address 127.31.131.65:43607)
0507 02:09:50.241933 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:50.241942 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0507 02:09:50.242021 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:09:50.242763 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:09:50.242766 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:09:50.243182 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:09:50.243428 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:50.243433 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:50.244324 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:09:50.244325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:09:50.244657 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:09:50.244658 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:09:50.244762 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:09:50.245372 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:09:50.245393 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:09:50.246808 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:09:50.248799 (+ 1991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:09:50.248805 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:09:50.248808 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:09:50.249084 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:09:50.249274 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:09:50.249275 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:09:50.249275 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:09:50.249321 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:09:50.249614 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:09:50.249619 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:09:50.249716 (+ 97us) client_negotiation.cc:769] Sending connection context
0507 02:09:50.249804 (+ 88us) client_negotiation.cc:240] Negotiation successful
0507 02:09:50.250029 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81,"mutex_wait_us":102,"thread_start_us":43,"threads_started":1}
W20240507 02:09:50.251019 14779 leader_election.cc:341] T a1d1ecdf24174716baceef2af183e91c P 1bec15577655406baa80dd159df72708 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3b07d0ce58504635ac42242bafc1719a (127.31.131.66:35163): Not found: Tablet not found: a1d1ecdf24174716baceef2af183e91c
W20240507 02:09:50.251128 14776 leader_election.cc:341] T a1d1ecdf24174716baceef2af183e91c P 1bec15577655406baa80dd159df72708 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cb007fe5498b4f9b95aea33b51a36185 (127.31.131.67:43055): Not found: Tablet not found: a1d1ecdf24174716baceef2af183e91c
I20240507 02:09:50.251192 14776 leader_election.cc:302] T a1d1ecdf24174716baceef2af183e91c P 1bec15577655406baa80dd159df72708 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1bec15577655406baa80dd159df72708; no voters: 3b07d0ce58504635ac42242bafc1719a, cb007fe5498b4f9b95aea33b51a36185
I20240507 02:09:50.251281 15163 raft_consensus.cc:2726] T a1d1ecdf24174716baceef2af183e91c P 1bec15577655406baa80dd159df72708 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 02:09:50.527740 15165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:50.527595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:36547 (local address 127.31.131.65:52407)
0507 02:09:50.527657 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:50.527693 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:36547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20240507 02:09:50.583142 15168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:50.582947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:36547 (local address 127.31.131.66:44073)
0507 02:09:50.583063 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:50.583101 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:36547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":66,"threads_started":1}
I20240507 02:09:50.695322 15169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:50.695120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:36547 (local address 127.31.131.67:36473)
0507 02:09:50.695243 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:50.695284 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:36547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":77,"threads_started":1}
I20240507 02:09:51.528394 15170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:51.528158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:36547 (local address 127.31.131.65:37471)
0507 02:09:51.528308 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:51.528348 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:36547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":66,"threads_started":1}
I20240507 02:09:51.583673 15171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:51.583503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:36547 (local address 127.31.131.66:37539)
0507 02:09:51.583607 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:51.583635 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:36547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":59,"threads_started":1}
I20240507 02:09:51.695875 15172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:51.695711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:36547 (local address 127.31.131.67:33299)
0507 02:09:51.695814 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:51.695839 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:36547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":63,"threads_started":1}
I20240507 02:09:52.332830 15173 raft_consensus.cc:482] T a1d1ecdf24174716baceef2af183e91c P 1bec15577655406baa80dd159df72708 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 02:09:52.332922 15173 raft_consensus.cc:504] T a1d1ecdf24174716baceef2af183e91c P 1bec15577655406baa80dd159df72708 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b07d0ce58504635ac42242bafc1719a" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 35163 } } peers { permanent_uuid: "cb007fe5498b4f9b95aea33b51a36185" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 43055 } } peers { permanent_uuid: "1bec15577655406baa80dd159df72708" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33661 } }
I20240507 02:09:52.333086 15173 leader_election.cc:288] T a1d1ecdf24174716baceef2af183e91c P 1bec15577655406baa80dd159df72708 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3b07d0ce58504635ac42242bafc1719a (127.31.131.66:35163), cb007fe5498b4f9b95aea33b51a36185 (127.31.131.67:43055)
W20240507 02:09:52.333408 14779 leader_election.cc:341] T a1d1ecdf24174716baceef2af183e91c P 1bec15577655406baa80dd159df72708 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3b07d0ce58504635ac42242bafc1719a (127.31.131.66:35163): Not found: Tablet not found: a1d1ecdf24174716baceef2af183e91c
W20240507 02:09:52.333487 14776 leader_election.cc:341] T a1d1ecdf24174716baceef2af183e91c P 1bec15577655406baa80dd159df72708 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cb007fe5498b4f9b95aea33b51a36185 (127.31.131.67:43055): Not found: Tablet not found: a1d1ecdf24174716baceef2af183e91c
I20240507 02:09:52.333556 14776 leader_election.cc:302] T a1d1ecdf24174716baceef2af183e91c P 1bec15577655406baa80dd159df72708 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1bec15577655406baa80dd159df72708; no voters: 3b07d0ce58504635ac42242bafc1719a, cb007fe5498b4f9b95aea33b51a36185
I20240507 02:09:52.333632 15173 raft_consensus.cc:2726] T a1d1ecdf24174716baceef2af183e91c P 1bec15577655406baa80dd159df72708 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 02:09:52.529018 15174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:52.528831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:36547 (local address 127.31.131.65:35167)
0507 02:09:52.528942 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:52.528971 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:36547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":65,"threads_started":1}
I20240507 02:09:52.584264 15175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:52.584068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:36547 (local address 127.31.131.66:48957)
0507 02:09:52.584191 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:52.584224 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:36547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":58,"threads_started":1}
I20240507 02:09:52.696451 15176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:52.696261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:36547 (local address 127.31.131.67:47259)
0507 02:09:52.696379 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:52.696412 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:36547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":68,"threads_started":1}
I20240507 02:09:53.529682 15177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:53.529484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:36547 (local address 127.31.131.65:46549)
0507 02:09:53.529604 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:53.529633 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:36547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":62,"threads_started":1}
I20240507 02:09:53.584857 15178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:53.584654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:36547 (local address 127.31.131.66:38079)
0507 02:09:53.584779 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:53.584813 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:36547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":63,"threads_started":1}
I20240507 02:09:53.697075 15179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:53.696840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:36547 (local address 127.31.131.67:48147)
0507 02:09:53.696996 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:53.697029 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:36547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":108,"threads_started":1}
I20240507 02:09:54.530319 15180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:54.530112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:36547 (local address 127.31.131.65:55703)
0507 02:09:54.530238 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:54.530271 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:36547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":69,"threads_started":1}
I20240507 02:09:54.585471 15181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:54.585262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:36547 (local address 127.31.131.66:51209)
0507 02:09:54.585392 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:54.585428 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:36547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":75,"threads_started":1}
I20240507 02:09:54.697597 15182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:54.697412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:36547 (local address 127.31.131.67:33545)
0507 02:09:54.697530 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:54.697560 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:36547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":70,"threads_started":1}
I20240507 02:09:55.168150 15183 raft_consensus.cc:482] T a1d1ecdf24174716baceef2af183e91c P 1bec15577655406baa80dd159df72708 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 02:09:55.168251 15183 raft_consensus.cc:504] T a1d1ecdf24174716baceef2af183e91c P 1bec15577655406baa80dd159df72708 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b07d0ce58504635ac42242bafc1719a" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 35163 } } peers { permanent_uuid: "cb007fe5498b4f9b95aea33b51a36185" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 43055 } } peers { permanent_uuid: "1bec15577655406baa80dd159df72708" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33661 } }
I20240507 02:09:55.168390 15183 leader_election.cc:288] T a1d1ecdf24174716baceef2af183e91c P 1bec15577655406baa80dd159df72708 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3b07d0ce58504635ac42242bafc1719a (127.31.131.66:35163), cb007fe5498b4f9b95aea33b51a36185 (127.31.131.67:43055)
W20240507 02:09:55.168747 14776 leader_election.cc:341] T a1d1ecdf24174716baceef2af183e91c P 1bec15577655406baa80dd159df72708 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cb007fe5498b4f9b95aea33b51a36185 (127.31.131.67:43055): Not found: Tablet not found: a1d1ecdf24174716baceef2af183e91c
W20240507 02:09:55.168831 14779 leader_election.cc:341] T a1d1ecdf24174716baceef2af183e91c P 1bec15577655406baa80dd159df72708 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3b07d0ce58504635ac42242bafc1719a (127.31.131.66:35163): Not found: Tablet not found: a1d1ecdf24174716baceef2af183e91c
I20240507 02:09:55.168933 14779 leader_election.cc:302] T a1d1ecdf24174716baceef2af183e91c P 1bec15577655406baa80dd159df72708 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1bec15577655406baa80dd159df72708; no voters: 3b07d0ce58504635ac42242bafc1719a, cb007fe5498b4f9b95aea33b51a36185
I20240507 02:09:55.169018 15183 raft_consensus.cc:2726] T a1d1ecdf24174716baceef2af183e91c P 1bec15577655406baa80dd159df72708 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 02:09:55.530953 15184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:55.530742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:36547 (local address 127.31.131.65:32783)
0507 02:09:55.530870 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:55.530903 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:36547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":72,"threads_started":1}
I20240507 02:09:55.586082 15185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:55.585861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:36547 (local address 127.31.131.66:52047)
0507 02:09:55.585991 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:55.586026 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:36547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":58,"threads_started":1}
I20240507 02:09:55.698202 15186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:55.697991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:36547 (local address 127.31.131.67:43509)
0507 02:09:55.698105 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:55.698156 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:36547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":61,"threads_started":1}
I20240507 02:09:56.531615 15187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:56.531405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:36547 (local address 127.31.131.65:38527)
0507 02:09:56.531529 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:56.531564 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:36547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":65,"threads_started":1}
I20240507 02:09:56.586680 15188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:56.586467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:36547 (local address 127.31.131.66:49847)
0507 02:09:56.586605 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:56.586637 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:36547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":56,"threads_started":1}
I20240507 02:09:56.698848 15189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:56.698642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:36547 (local address 127.31.131.67:46955)
0507 02:09:56.698762 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:56.698804 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:36547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":71,"threads_started":1}
I20240507 02:09:57.532289 15190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:57.532056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:36547 (local address 127.31.131.65:56707)
0507 02:09:57.532190 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:57.532227 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:36547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":77,"threads_started":1}
I20240507 02:09:57.587281 15191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:57.587063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:36547 (local address 127.31.131.66:41471)
0507 02:09:57.587204 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:57.587235 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:36547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":84,"threads_started":1}
I20240507 02:09:57.699388 15192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:57.699193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:36547 (local address 127.31.131.67:33959)
0507 02:09:57.699308 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:57.699341 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:36547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":73,"threads_started":1}
I20240507 02:09:58.532927 15193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:58.532711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:36547 (local address 127.31.131.65:44169)
0507 02:09:58.532843 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:58.532877 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:36547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":68,"threads_started":1}
I20240507 02:09:58.587874 15194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:58.587656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:36547 (local address 127.31.131.66:43393)
0507 02:09:58.587795 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:58.587829 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:36547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":77,"threads_started":1}
I20240507 02:09:58.699901 15195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:58.699721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:36547 (local address 127.31.131.67:47637)
0507 02:09:58.699826 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:58.699857 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:36547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":58,"threads_started":1}
I20240507 02:09:59.510787 15196 raft_consensus.cc:482] T a1d1ecdf24174716baceef2af183e91c P 1bec15577655406baa80dd159df72708 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 02:09:59.510877 15196 raft_consensus.cc:504] T a1d1ecdf24174716baceef2af183e91c P 1bec15577655406baa80dd159df72708 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b07d0ce58504635ac42242bafc1719a" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 35163 } } peers { permanent_uuid: "cb007fe5498b4f9b95aea33b51a36185" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 43055 } } peers { permanent_uuid: "1bec15577655406baa80dd159df72708" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33661 } }
I20240507 02:09:59.511009 15196 leader_election.cc:288] T a1d1ecdf24174716baceef2af183e91c P 1bec15577655406baa80dd159df72708 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3b07d0ce58504635ac42242bafc1719a (127.31.131.66:35163), cb007fe5498b4f9b95aea33b51a36185 (127.31.131.67:43055)
W20240507 02:09:59.511341 14776 leader_election.cc:341] T a1d1ecdf24174716baceef2af183e91c P 1bec15577655406baa80dd159df72708 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cb007fe5498b4f9b95aea33b51a36185 (127.31.131.67:43055): Not found: Tablet not found: a1d1ecdf24174716baceef2af183e91c
W20240507 02:09:59.511438 14779 leader_election.cc:341] T a1d1ecdf24174716baceef2af183e91c P 1bec15577655406baa80dd159df72708 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3b07d0ce58504635ac42242bafc1719a (127.31.131.66:35163): Not found: Tablet not found: a1d1ecdf24174716baceef2af183e91c
I20240507 02:09:59.511482 14779 leader_election.cc:302] T a1d1ecdf24174716baceef2af183e91c P 1bec15577655406baa80dd159df72708 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1bec15577655406baa80dd159df72708; no voters: 3b07d0ce58504635ac42242bafc1719a, cb007fe5498b4f9b95aea33b51a36185
I20240507 02:09:59.511543 15196 raft_consensus.cc:2726] T a1d1ecdf24174716baceef2af183e91c P 1bec15577655406baa80dd159df72708 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 02:09:59.533564 15197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:59.533310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:36547 (local address 127.31.131.65:36807)
0507 02:09:59.533464 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:59.533503 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:36547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":70,"threads_started":1}
I20240507 02:09:59.588497 15198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:59.588257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:36547 (local address 127.31.131.66:39717)
0507 02:09:59.588414 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:59.588450 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:36547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":82,"threads_started":1}
I20240507 02:09:59.700479 15199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:09:59.700274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:36547 (local address 127.31.131.67:44993)
0507 02:09:59.700391 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0507 02:09:59.700430 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:36547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":59,"threads_started":1}
I20240507 02:10:00.534168 15200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:00.533968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:36547 (local address 127.31.131.65:60973)
0507 02:10:00.534088 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:00.534121 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:36547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":65,"threads_started":1}
I20240507 02:10:00.589044 15201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:00.588866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:36547 (local address 127.31.131.66:50657)
0507 02:10:00.588971 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:00.589001 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:36547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":57,"threads_started":1}
I20240507 02:10:00.701081 15202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:00.700884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:36547 (local address 127.31.131.67:51849)
0507 02:10:00.700996 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:00.701033 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:36547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":62,"threads_started":1}
I20240507 02:10:01.534919 15203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:01.534638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:36547 (local address 127.31.131.65:60895)
0507 02:10:01.534807 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:01.534856 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:36547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":85,"threads_started":1}
I20240507 02:10:01.589782 15204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:01.589548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:36547 (local address 127.31.131.66:49033)
0507 02:10:01.589692 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:01.589735 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:36547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":68,"threads_started":1}
I20240507 02:10:01.701736 15205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:01.701509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:36547 (local address 127.31.131.67:33931)
0507 02:10:01.701638 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:01.701682 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:36547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":64,"threads_started":1}
I20240507 02:10:02.535615 15206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:02.535398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:36547 (local address 127.31.131.65:37619)
0507 02:10:02.535530 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:02.535567 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:36547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":71,"threads_started":1}
I20240507 02:10:02.590391 15207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:02.590206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:36547 (local address 127.31.131.66:47475)
0507 02:10:02.590319 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:02.590348 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:36547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":58,"threads_started":1}
I20240507 02:10:02.702432 15208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:02.702205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:36547 (local address 127.31.131.67:57083)
0507 02:10:02.702336 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:02.702381 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:36547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":74,"threads_started":1}
I20240507 02:10:03.536362 15209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:03.536106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:36547 (local address 127.31.131.65:47609)
0507 02:10:03.536245 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:03.536300 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:36547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":68,"threads_started":1}
I20240507 02:10:03.591023 15210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:03.590810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:36547 (local address 127.31.131.66:40541)
0507 02:10:03.590939 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:03.590978 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:36547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":60,"threads_started":1}
I20240507 02:10:03.703121 15211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:03.702906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:36547 (local address 127.31.131.67:44403)
0507 02:10:03.703040 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:03.703075 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:36547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":62,"threads_started":1}
I20240507 02:10:03.705720 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 14731
I20240507 02:10:03.710023 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 14894
I20240507 02:10:03.713753 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 15027
2024-05-07T02:10:03Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (17725 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 07 02:10:03 dist-test-slave-v0bg krb5kdc[15214](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 07 02:10:03 dist-test-slave-v0bg krb5kdc[15214](info): set up 2 sockets
May 07 02:10:03 dist-test-slave-v0bg krb5kdc[15214](info): commencing operation
krb5kdc: starting...
W20240507 02:10:04.599277 32269 mini_kdc.cc:121] Time spent starting KDC: real 0.868s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/krb5kdc/test-user.keytab.
May 07 02:10:04 dist-test-slave-v0bg krb5kdc[15214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047804, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-07T02:10:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-07T02:10:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.131.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab.
Entry for principal kudu/127.31.131.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab.
I20240507 02:10:04.698472 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:35343
--webserver_interface=127.31.131.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab
--principal=kudu/127.31.131.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:43653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.131.126:35343
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:10:04.772058 15227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:10:04.772230 15227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:10:04.772250 15227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:10:04.773574 15227 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20240507 02:10:04.773615 15227 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20240507 02:10:04.773626 15227 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20240507 02:10:04.773636 15227 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240507 02:10:04.773646 15227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:10:04.773679 15227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:10:04.773695 15227 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240507 02:10:04.773725 15227 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240507 02:10:04.775342 15227 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:43653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.31.131.126:35343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab
--principal=kudu/127.31.131.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:35343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.131.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15227
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:10:04.775533 15227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:10:04.775698 15227 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:10:04.779213 15235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:10:04.779301 15233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:10:04.779322 15227 server_base.cc:1034] running on GCE node
W20240507 02:10:04.779417 15232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:10:04.779721 15227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:10:04.779924 15227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:10:04.781046 15227 hybrid_clock.cc:648] HybridClock initialized: now 1715047804781029 us; error 31 us; skew 500 ppm
May 07 02:10:04 dist-test-slave-v0bg krb5kdc[15214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047804, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:10:04.787496 15227 init.cc:377] Logged in from keytab as kudu/127.31.131.126@KRBTEST.COM (short username kudu)
I20240507 02:10:04.787884 15227 webserver.cc:460] Webserver started at http://127.31.131.126:41271/ using document root <none> and password file <none>
I20240507 02:10:04.788054 15227 fs_manager.cc:359] Metadata directory not provided
I20240507 02:10:04.788093 15227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:10:04.788192 15227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:10:04.789112 15227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/master-0/data/instance:
uuid: "dec8ad1f575d48a59f9845b310f4af5b"
format_stamp: "Formatted at 2024-05-07 02:10:04 on dist-test-slave-v0bg"
server_key: "0b9180c36ad370f362912ab9d5c00dbc"
server_key_iv: "e25f55c88aa80c06a136d70722054126"
server_key_version: "encryptionkey@0"
I20240507 02:10:04.789384 15227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/master-0/wal/instance:
uuid: "dec8ad1f575d48a59f9845b310f4af5b"
format_stamp: "Formatted at 2024-05-07 02:10:04 on dist-test-slave-v0bg"
server_key: "0b9180c36ad370f362912ab9d5c00dbc"
server_key_iv: "e25f55c88aa80c06a136d70722054126"
server_key_version: "encryptionkey@0"
I20240507 02:10:04.790452 15227 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240507 02:10:04.791081 15242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:10:04.791220 15227 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240507 02:10:04.791290 15227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/master-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/master-0/wal
uuid: "dec8ad1f575d48a59f9845b310f4af5b"
format_stamp: "Formatted at 2024-05-07 02:10:04 on dist-test-slave-v0bg"
server_key: "0b9180c36ad370f362912ab9d5c00dbc"
server_key_iv: "e25f55c88aa80c06a136d70722054126"
server_key_version: "encryptionkey@0"
I20240507 02:10:04.791344 15227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:10:04.813272 15227 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:10:04.814898 15227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:10:04.815009 15227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:10:04.818298 15227 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.126:35343
I20240507 02:10:04.818332 15294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.126:35343 every 8 connection(s)
I20240507 02:10:04.818640 15227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
I20240507 02:10:04.819316 15295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:10:04.821334 15295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dec8ad1f575d48a59f9845b310f4af5b: Bootstrap starting.
I20240507 02:10:04.821799 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 15227
I20240507 02:10:04.821897 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/master-0/wal/instance
I20240507 02:10:04.821931 15295 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dec8ad1f575d48a59f9845b310f4af5b: Neither blocks nor log segments found. Creating new log.
I20240507 02:10:04.822057 32269 external_mini_cluster.cc:1428] Setting key 21bbaae940f95ad948bb0093ffea2796
I20240507 02:10:04.822196 15295 log.cc:824] T 00000000000000000000000000000000 P dec8ad1f575d48a59f9845b310f4af5b: Log is configured to *not* fsync() on all Append() calls
I20240507 02:10:04.822885 15295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dec8ad1f575d48a59f9845b310f4af5b: No bootstrap required, opened a new log
I20240507 02:10:04.824414 15295 raft_consensus.cc:348] T 00000000000000000000000000000000 P dec8ad1f575d48a59f9845b310f4af5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec8ad1f575d48a59f9845b310f4af5b" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 35343 } }
I20240507 02:10:04.824527 15295 raft_consensus.cc:374] T 00000000000000000000000000000000 P dec8ad1f575d48a59f9845b310f4af5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:10:04.824549 15295 raft_consensus.cc:729] T 00000000000000000000000000000000 P dec8ad1f575d48a59f9845b310f4af5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec8ad1f575d48a59f9845b310f4af5b, State: Initialized, Role: FOLLOWER
I20240507 02:10:04.824638 15295 consensus_queue.cc:260] T 00000000000000000000000000000000 P dec8ad1f575d48a59f9845b310f4af5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec8ad1f575d48a59f9845b310f4af5b" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 35343 } }
I20240507 02:10:04.824699 15295 raft_consensus.cc:388] T 00000000000000000000000000000000 P dec8ad1f575d48a59f9845b310f4af5b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 02:10:04.824728 15295 raft_consensus.cc:482] T 00000000000000000000000000000000 P dec8ad1f575d48a59f9845b310f4af5b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 02:10:04.824771 15295 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dec8ad1f575d48a59f9845b310f4af5b [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:10:04.825244 15295 raft_consensus.cc:504] T 00000000000000000000000000000000 P dec8ad1f575d48a59f9845b310f4af5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec8ad1f575d48a59f9845b310f4af5b" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 35343 } }
I20240507 02:10:04.825345 15295 leader_election.cc:302] T 00000000000000000000000000000000 P dec8ad1f575d48a59f9845b310f4af5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dec8ad1f575d48a59f9845b310f4af5b; no voters:
I20240507 02:10:04.825486 15295 leader_election.cc:288] T 00000000000000000000000000000000 P dec8ad1f575d48a59f9845b310f4af5b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240507 02:10:04.825534 15300 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dec8ad1f575d48a59f9845b310f4af5b [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:10:04.825686 15295 sys_catalog.cc:564] T 00000000000000000000000000000000 P dec8ad1f575d48a59f9845b310f4af5b [sys.catalog]: configured and running, proceeding with master startup.
I20240507 02:10:04.825711 15300 raft_consensus.cc:686] T 00000000000000000000000000000000 P dec8ad1f575d48a59f9845b310f4af5b [term 1 LEADER]: Becoming Leader. State: Replica: dec8ad1f575d48a59f9845b310f4af5b, State: Running, Role: LEADER
I20240507 02:10:04.825853 15300 consensus_queue.cc:237] T 00000000000000000000000000000000 P dec8ad1f575d48a59f9845b310f4af5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec8ad1f575d48a59f9845b310f4af5b" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 35343 } }
I20240507 02:10:04.826169 15302 sys_catalog.cc:455] T 00000000000000000000000000000000 P dec8ad1f575d48a59f9845b310f4af5b [sys.catalog]: SysCatalogTable state changed. Reason: New leader dec8ad1f575d48a59f9845b310f4af5b. Latest consensus state: current_term: 1 leader_uuid: "dec8ad1f575d48a59f9845b310f4af5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec8ad1f575d48a59f9845b310f4af5b" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 35343 } } }
I20240507 02:10:04.826233 15302 sys_catalog.cc:458] T 00000000000000000000000000000000 P dec8ad1f575d48a59f9845b310f4af5b [sys.catalog]: This master's current role is: LEADER
I20240507 02:10:04.826294 15301 sys_catalog.cc:455] T 00000000000000000000000000000000 P dec8ad1f575d48a59f9845b310f4af5b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dec8ad1f575d48a59f9845b310f4af5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec8ad1f575d48a59f9845b310f4af5b" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 35343 } } }
I20240507 02:10:04.826349 15301 sys_catalog.cc:458] T 00000000000000000000000000000000 P dec8ad1f575d48a59f9845b310f4af5b [sys.catalog]: This master's current role is: LEADER
I20240507 02:10:04.826599 15307 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 02:10:04.827064 15307 catalog_manager.cc:1484] Initializing Kudu cluster ID...
May 07 02:10:04 dist-test-slave-v0bg krb5kdc[15214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047804, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:10:04.828470 15307 catalog_manager.cc:1347] Generated new cluster ID: 69cbc585993f46de82c1ce606effdf61
I20240507 02:10:04.828518 15307 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240507 02:10:04.831219 15299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:04.822497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47384 (local address 127.31.131.126:35343)
0507 02:10:04.822646 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0507 02:10:04.822657 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0507 02:10:04.822690 (+ 33us) server_negotiation.cc:407] Connection header received
0507 02:10:04.823146 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:10:04.823152 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:10:04.823194 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:10:04.823304 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:10:04.824198 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:10:04.825208 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:10:04.825951 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:10:04.826182 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:10:04.828477 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:10:04.828499 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:10:04.828499 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:10:04.828518 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:10:04.830016 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:10:04.830297 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:10:04.830299 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:10:04.830300 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:10:04.830342 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:10:04.830478 (+ 136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:10:04.830479 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:10:04.830479 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:10:04.830691 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:10:04.830818 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0507 02:10:04.830930 (+ 112us) server_negotiation.cc:299] Negotiation successful
0507 02:10:04.831039 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":79,"threads_started":1}
I20240507 02:10:04.836700 15307 catalog_manager.cc:1370] Generated new certificate authority record
I20240507 02:10:04.837360 15307 catalog_manager.cc:1504] Loading token signing keys...
I20240507 02:10:04.845429 15307 catalog_manager.cc:5857] T 00000000000000000000000000000000 P dec8ad1f575d48a59f9845b310f4af5b: Generated new TSK 0
I20240507 02:10:04.845613 15307 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.131.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab.
Entry for principal kudu/127.31.131.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab.
I20240507 02:10:04.869313 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.65:0
--local_ip_for_outbound_sockets=127.31.131.65
--webserver_interface=127.31.131.65
--webserver_port=0
--tserver_master_addrs=127.31.131.126:35343
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab
--principal=kudu/127.31.131.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:43653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:10:04.945252 15321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:10:04.945434 15321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:10:04.945462 15321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:10:04.946779 15321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:10:04.946821 15321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:10:04.946853 15321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.65
I20240507 02:10:04.948418 15321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:43653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab
--principal=kudu/127.31.131.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.131.65
--webserver_port=0
--tserver_master_addrs=127.31.131.126:35343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.65
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:10:04.948628 15321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:10:04.948808 15321 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240507 02:10:04.953001 15321 server_base.cc:1034] running on GCE node
W20240507 02:10:04.952961 15329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:10:04.953104 15327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:10:04.953094 15326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:10:04.953372 15321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:10:04.953568 15321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:10:04.954699 15321 hybrid_clock.cc:648] HybridClock initialized: now 1715047804954686 us; error 29 us; skew 500 ppm
May 07 02:10:04 dist-test-slave-v0bg krb5kdc[15214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047804, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:10:04.957199 15321 init.cc:377] Logged in from keytab as kudu/127.31.131.65@KRBTEST.COM (short username kudu)
I20240507 02:10:04.957671 15321 webserver.cc:460] Webserver started at http://127.31.131.65:40345/ using document root <none> and password file <none>
I20240507 02:10:04.957875 15321 fs_manager.cc:359] Metadata directory not provided
I20240507 02:10:04.957931 15321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:10:04.958045 15321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:10:04.959337 15321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-0/data/instance:
uuid: "0d6e5d3dac474d61a6960acc0b360dbc"
format_stamp: "Formatted at 2024-05-07 02:10:04 on dist-test-slave-v0bg"
server_key: "5eeb8e1ff74dba96a2a309487f9cf3ea"
server_key_iv: "33df10cbc886a1a47eeff61fa4082c42"
server_key_version: "encryptionkey@0"
I20240507 02:10:04.959707 15321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-0/wal/instance:
uuid: "0d6e5d3dac474d61a6960acc0b360dbc"
format_stamp: "Formatted at 2024-05-07 02:10:04 on dist-test-slave-v0bg"
server_key: "5eeb8e1ff74dba96a2a309487f9cf3ea"
server_key_iv: "33df10cbc886a1a47eeff61fa4082c42"
server_key_version: "encryptionkey@0"
I20240507 02:10:04.961160 15321 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240507 02:10:04.961875 15336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:10:04.962035 15321 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240507 02:10:04.962126 15321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-0/wal
uuid: "0d6e5d3dac474d61a6960acc0b360dbc"
format_stamp: "Formatted at 2024-05-07 02:10:04 on dist-test-slave-v0bg"
server_key: "5eeb8e1ff74dba96a2a309487f9cf3ea"
server_key_iv: "33df10cbc886a1a47eeff61fa4082c42"
server_key_version: "encryptionkey@0"
I20240507 02:10:04.962193 15321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:10:04.972085 15321 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:10:04.974026 15321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:10:04.974150 15321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:10:04.974351 15321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:10:04.974715 15321 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:10:04.974758 15321 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:10:04.974787 15321 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:10:04.974810 15321 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:10:04.981177 15321 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.65:43457
I20240507 02:10:04.981225 15449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.65:43457 every 8 connection(s)
I20240507 02:10:04.981501 15321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
I20240507 02:10:04.983525 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 15321
I20240507 02:10:04.983644 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-0/wal/instance
I20240507 02:10:04.983824 32269 external_mini_cluster.cc:1428] Setting key 74c1a435dd6790bc8889236255b6d9c0
May 07 02:10:04 dist-test-slave-v0bg krb5kdc[15214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047804, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.65@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:10:04.990051 15299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:04.982169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:50261 (local address 127.31.131.126:35343)
0507 02:10:04.982210 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0507 02:10:04.982212 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:10:04.982378 (+ 166us) server_negotiation.cc:407] Connection header received
0507 02:10:04.982862 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:10:04.982864 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:10:04.982875 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:10:04.982950 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:10:04.984381 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:10:04.985039 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:10:04.985771 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:10:04.985940 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:10:04.987683 (+ 1743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:10:04.987686 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:10:04.987687 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:10:04.987692 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:10:04.988998 (+ 1306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:10:04.989318 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:10:04.989319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:10:04.989320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:10:04.989341 (+ 21us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:10:04.989490 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:10:04.989491 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:10:04.989491 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:10:04.989578 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:10:04.989633 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0507 02:10:04.989859 (+ 226us) server_negotiation.cc:299] Negotiation successful
0507 02:10:04.989951 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20240507 02:10:04.990131 15452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:04.982133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:35343 (local address 127.31.131.65:50261)
0507 02:10:04.982269 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:04.982298 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0507 02:10:04.982766 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:10:04.983000 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:10:04.983003 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:10:04.983283 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:10:04.984018 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:10:04.984024 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:10:04.985139 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:10:04.985140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:10:04.985651 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:10:04.985654 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:10:04.985818 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:10:04.986305 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:10:04.986320 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:10:04.987598 (+ 1278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:10:04.989070 (+ 1472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:10:04.989075 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:10:04.989078 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:10:04.989256 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:10:04.989390 (+ 134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:10:04.989391 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:10:04.989391 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:10:04.989443 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:10:04.989630 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:10:04.989635 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:10:04.989743 (+ 108us) client_negotiation.cc:769] Sending connection context
0507 02:10:04.989833 (+ 90us) client_negotiation.cc:240] Negotiation successful
0507 02:10:04.989956 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":60,"threads_started":1}
WARNING: no policy specified for kudu/127.31.131.66@KRBTEST.COM; defaulting to no policy
I20240507 02:10:04.990617 15450 heartbeater.cc:347] Connected to a master server at 127.31.131.126:35343
I20240507 02:10:04.990701 15450 heartbeater.cc:464] Registering TS with master...
I20240507 02:10:04.990885 15450 heartbeater.cc:511] Master 127.31.131.126:35343 requested a full tablet report, sending...
I20240507 02:10:04.991371 15259 ts_manager.cc:194] Registered new tserver with Master: 0d6e5d3dac474d61a6960acc0b360dbc (127.31.131.65:43457)
I20240507 02:10:04.991945 15259 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.131.65@KRBTEST.COM'} at 127.31.131.65:50261
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab.
Entry for principal kudu/127.31.131.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab.
I20240507 02:10:05.004889 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.66:0
--local_ip_for_outbound_sockets=127.31.131.66
--webserver_interface=127.31.131.66
--webserver_port=0
--tserver_master_addrs=127.31.131.126:35343
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab
--principal=kudu/127.31.131.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:43653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:10:05.079111 15455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:10:05.079280 15455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:10:05.079300 15455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:10:05.080521 15455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:10:05.080561 15455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:10:05.080586 15455 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.66
I20240507 02:10:05.082077 15455 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:43653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab
--principal=kudu/127.31.131.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.131.66
--webserver_port=0
--tserver_master_addrs=127.31.131.126:35343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.66
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:10:05.082264 15455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:10:05.082427 15455 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:10:05.086040 15461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:10:05.086143 15455 server_base.cc:1034] running on GCE node
W20240507 02:10:05.086189 15463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:10:05.086644 15460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:10:05.086814 15455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:10:05.087004 15455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:10:05.088128 15455 hybrid_clock.cc:648] HybridClock initialized: now 1715047805088106 us; error 36 us; skew 500 ppm
May 07 02:10:05 dist-test-slave-v0bg krb5kdc[15214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047805, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:10:05.090232 15455 init.cc:377] Logged in from keytab as kudu/127.31.131.66@KRBTEST.COM (short username kudu)
I20240507 02:10:05.090618 15455 webserver.cc:460] Webserver started at http://127.31.131.66:32811/ using document root <none> and password file <none>
I20240507 02:10:05.090824 15455 fs_manager.cc:359] Metadata directory not provided
I20240507 02:10:05.090864 15455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:10:05.090951 15455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:10:05.091845 15455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-1/data/instance:
uuid: "c5a8b5ba9bf94f4aa922d3d5307406b6"
format_stamp: "Formatted at 2024-05-07 02:10:05 on dist-test-slave-v0bg"
server_key: "da659846df383f31f89105bcdf6de9d0"
server_key_iv: "22baf740d3e4b3c8bcee96872b10d9c9"
server_key_version: "encryptionkey@0"
I20240507 02:10:05.092114 15455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-1/wal/instance:
uuid: "c5a8b5ba9bf94f4aa922d3d5307406b6"
format_stamp: "Formatted at 2024-05-07 02:10:05 on dist-test-slave-v0bg"
server_key: "da659846df383f31f89105bcdf6de9d0"
server_key_iv: "22baf740d3e4b3c8bcee96872b10d9c9"
server_key_version: "encryptionkey@0"
I20240507 02:10:05.093135 15455 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240507 02:10:05.093740 15470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:10:05.093881 15455 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240507 02:10:05.093950 15455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-1/wal
uuid: "c5a8b5ba9bf94f4aa922d3d5307406b6"
format_stamp: "Formatted at 2024-05-07 02:10:05 on dist-test-slave-v0bg"
server_key: "da659846df383f31f89105bcdf6de9d0"
server_key_iv: "22baf740d3e4b3c8bcee96872b10d9c9"
server_key_version: "encryptionkey@0"
I20240507 02:10:05.094007 15455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:10:05.108983 15455 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:10:05.110399 15455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:10:05.110495 15455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:10:05.110695 15455 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:10:05.110972 15455 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:10:05.111003 15455 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:10:05.111029 15455 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:10:05.111047 15455 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:10:05.115983 15455 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.66:38643
I20240507 02:10:05.116035 15583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.66:38643 every 8 connection(s)
I20240507 02:10:05.116290 15455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
I20240507 02:10:05.118633 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 15455
I20240507 02:10:05.118729 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-1/wal/instance
I20240507 02:10:05.118893 32269 external_mini_cluster.cc:1428] Setting key f04fb26cf512151bd2bb2f96f547c3fa
May 07 02:10:05 dist-test-slave-v0bg krb5kdc[15214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047805, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.66@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:10:05.124135 15299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:05.116869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.66:56037 (local address 127.31.131.126:35343)
0507 02:10:05.116942 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0507 02:10:05.116944 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:10:05.117054 (+ 110us) server_negotiation.cc:407] Connection header received
0507 02:10:05.117489 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:10:05.117490 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:10:05.117499 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:10:05.117545 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:10:05.118425 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:10:05.118996 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:10:05.119579 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:10:05.119733 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:10:05.121625 (+ 1892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:10:05.121628 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:10:05.121629 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:10:05.121635 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:10:05.122999 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:10:05.123420 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:10:05.123420 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:10:05.123421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:10:05.123437 (+ 16us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:10:05.123613 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:10:05.123613 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:10:05.123614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:10:05.123691 (+ 77us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:10:05.123731 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0507 02:10:05.124008 (+ 277us) server_negotiation.cc:299] Negotiation successful
0507 02:10:05.124066 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20240507 02:10:05.124323 15586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:05.116835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:35343 (local address 127.31.131.66:56037)
0507 02:10:05.116952 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:05.116976 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0507 02:10:05.117409 (+ 433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:10:05.117591 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:10:05.117594 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:10:05.117866 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:10:05.118354 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:10:05.118358 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:10:05.119070 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:10:05.119071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:10:05.119509 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:10:05.119511 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:10:05.119639 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:10:05.120151 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:10:05.120168 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:10:05.121534 (+ 1366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:10:05.123077 (+ 1543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:10:05.123083 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:10:05.123086 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:10:05.123349 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:10:05.123487 (+ 138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:10:05.123488 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:10:05.123489 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:10:05.123561 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:10:05.123741 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:10:05.123745 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:10:05.123885 (+ 140us) client_negotiation.cc:769] Sending connection context
0507 02:10:05.124003 (+ 118us) client_negotiation.cc:240] Negotiation successful
0507 02:10:05.124131 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":66,"threads_started":1}
I20240507 02:10:05.124749 15584 heartbeater.cc:347] Connected to a master server at 127.31.131.126:35343
I20240507 02:10:05.124830 15584 heartbeater.cc:464] Registering TS with master...
I20240507 02:10:05.125032 15584 heartbeater.cc:511] Master 127.31.131.126:35343 requested a full tablet report, sending...
I20240507 02:10:05.125396 15259 ts_manager.cc:194] Registered new tserver with Master: c5a8b5ba9bf94f4aa922d3d5307406b6 (127.31.131.66:38643)
I20240507 02:10:05.125758 15259 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.131.66@KRBTEST.COM'} at 127.31.131.66:56037
WARNING: no policy specified for kudu/127.31.131.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab.
Entry for principal kudu/127.31.131.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab.
I20240507 02:10:05.139412 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.67:0
--local_ip_for_outbound_sockets=127.31.131.67
--webserver_interface=127.31.131.67
--webserver_port=0
--tserver_master_addrs=127.31.131.126:35343
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab
--principal=kudu/127.31.131.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:43653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:10:05.216185 15589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:10:05.216367 15589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:10:05.216388 15589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:10:05.217619 15589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:10:05.217660 15589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:10:05.217684 15589 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.67
I20240507 02:10:05.219195 15589 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:43653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab
--principal=kudu/127.31.131.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.131.67
--webserver_port=0
--tserver_master_addrs=127.31.131.126:35343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.67
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:10:05.219353 15589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:10:05.219508 15589 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:10:05.223217 15595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:10:05.223233 15594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:10:05.223374 15597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:10:05.223515 15589 server_base.cc:1034] running on GCE node
I20240507 02:10:05.223680 15589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:10:05.223868 15589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:10:05.224988 15589 hybrid_clock.cc:648] HybridClock initialized: now 1715047805224978 us; error 26 us; skew 500 ppm
May 07 02:10:05 dist-test-slave-v0bg krb5kdc[15214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047805, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:10:05.227332 15589 init.cc:377] Logged in from keytab as kudu/127.31.131.67@KRBTEST.COM (short username kudu)
I20240507 02:10:05.227691 15589 webserver.cc:460] Webserver started at http://127.31.131.67:46439/ using document root <none> and password file <none>
I20240507 02:10:05.227831 15589 fs_manager.cc:359] Metadata directory not provided
I20240507 02:10:05.227864 15589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:10:05.227941 15589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:10:05.228839 15589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-2/data/instance:
uuid: "da19d23f7add4b50bf53330de753ce77"
format_stamp: "Formatted at 2024-05-07 02:10:05 on dist-test-slave-v0bg"
server_key: "0bad5911b238d5d777b5e092a8e2d14b"
server_key_iv: "ddb757c23556d4f449a3d2b79f605e33"
server_key_version: "encryptionkey@0"
I20240507 02:10:05.229090 15589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-2/wal/instance:
uuid: "da19d23f7add4b50bf53330de753ce77"
format_stamp: "Formatted at 2024-05-07 02:10:05 on dist-test-slave-v0bg"
server_key: "0bad5911b238d5d777b5e092a8e2d14b"
server_key_iv: "ddb757c23556d4f449a3d2b79f605e33"
server_key_version: "encryptionkey@0"
I20240507 02:10:05.230099 15589 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240507 02:10:05.230670 15604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:10:05.230823 15589 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240507 02:10:05.230893 15589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-2/wal
uuid: "da19d23f7add4b50bf53330de753ce77"
format_stamp: "Formatted at 2024-05-07 02:10:05 on dist-test-slave-v0bg"
server_key: "0bad5911b238d5d777b5e092a8e2d14b"
server_key_iv: "ddb757c23556d4f449a3d2b79f605e33"
server_key_version: "encryptionkey@0"
I20240507 02:10:05.230943 15589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:10:05.250485 15589 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:10:05.252645 15589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:10:05.252748 15589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:10:05.252941 15589 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:10:05.253279 15589 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:10:05.253315 15589 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:10:05.253350 15589 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:10:05.253384 15589 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:10:05.259217 15589 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.67:42387
I20240507 02:10:05.259276 15717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.67:42387 every 8 connection(s)
I20240507 02:10:05.259554 15589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
I20240507 02:10:05.263303 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 15589
I20240507 02:10:05.263418 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-2/wal/instance
I20240507 02:10:05.263585 32269 external_mini_cluster.cc:1428] Setting key 2187733b9812fffd5d9fcab882c8fb61
May 07 02:10:05 dist-test-slave-v0bg krb5kdc[15214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047805, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.67@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:10:05.267150 15299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:05.260185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.67:58117 (local address 127.31.131.126:35343)
0507 02:10:05.260253 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0507 02:10:05.260255 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:10:05.260371 (+ 116us) server_negotiation.cc:407] Connection header received
0507 02:10:05.260810 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:10:05.260811 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:10:05.260821 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:10:05.260881 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:10:05.261768 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:10:05.262249 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:10:05.262770 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:10:05.262880 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:10:05.264807 (+ 1927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:10:05.264810 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:10:05.264810 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:10:05.264815 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:10:05.266063 (+ 1248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:10:05.266347 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:10:05.266348 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:10:05.266348 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:10:05.266365 (+ 17us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:10:05.266574 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:10:05.266576 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:10:05.266576 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:10:05.266679 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:10:05.266733 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0507 02:10:05.266959 (+ 226us) server_negotiation.cc:299] Negotiation successful
0507 02:10:05.267045 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20240507 02:10:05.267222 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:05.260127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:35343 (local address 127.31.131.67:58117)
0507 02:10:05.260267 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:05.260295 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0507 02:10:05.260729 (+ 434us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:10:05.260932 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:10:05.260935 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:10:05.261197 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:10:05.261699 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:10:05.261703 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:10:05.262307 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:10:05.262308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:10:05.262714 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:10:05.262716 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:10:05.262807 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:10:05.263261 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:10:05.263282 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:10:05.264737 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:10:05.266119 (+ 1382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:10:05.266123 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:10:05.266125 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:10:05.266294 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:10:05.266430 (+ 136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:10:05.266432 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:10:05.266432 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:10:05.266495 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:10:05.266723 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:10:05.266726 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:10:05.266824 (+ 98us) client_negotiation.cc:769] Sending connection context
0507 02:10:05.266914 (+ 90us) client_negotiation.cc:240] Negotiation successful
0507 02:10:05.267048 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":79,"threads_started":1}
I20240507 02:10:05.267611 15718 heartbeater.cc:347] Connected to a master server at 127.31.131.126:35343
I20240507 02:10:05.267691 15718 heartbeater.cc:464] Registering TS with master...
I20240507 02:10:05.267872 15718 heartbeater.cc:511] Master 127.31.131.126:35343 requested a full tablet report, sending...
I20240507 02:10:05.268204 15259 ts_manager.cc:194] Registered new tserver with Master: da19d23f7add4b50bf53330de753ce77 (127.31.131.67:42387)
I20240507 02:10:05.268548 15259 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.131.67@KRBTEST.COM'} at 127.31.131.67:58117
I20240507 02:10:05.275152 32269 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240507 02:10:05.280962 15299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:05.276312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47390 (local address 127.31.131.126:35343)
0507 02:10:05.276364 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0507 02:10:05.276365 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:10:05.276455 (+ 90us) server_negotiation.cc:407] Connection header received
0507 02:10:05.276516 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:10:05.276517 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:10:05.276527 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:10:05.276574 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:10:05.277208 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:10:05.277663 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:10:05.278178 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:10:05.278329 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:10:05.279165 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:10:05.279168 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:10:05.279169 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:10:05.279174 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:10:05.280133 (+ 959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:10:05.280385 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:10:05.280386 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:10:05.280386 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:10:05.280406 (+ 20us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:10:05.280523 (+ 117us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:10:05.280524 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:10:05.280524 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:10:05.280625 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:10:05.280712 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0507 02:10:05.280801 (+ 89us) server_negotiation.cc:299] Negotiation successful
0507 02:10:05.280874 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20240507 02:10:05.282231 15259 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47390:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 02:10:05.283622 15259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 02:10:05.288374 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:05.285580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.65:43457 (local address 127.0.0.1:38244)
0507 02:10:05.285719 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:05.285728 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0507 02:10:05.285780 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:10:05.286101 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:10:05.286102 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:10:05.286104 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:10:05.286264 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:10:05.286266 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:10:05.287269 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:10:05.287270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:10:05.288012 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:10:05.288013 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:10:05.288083 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:10:05.288118 (+ 35us) client_negotiation.cc:769] Sending connection context
0507 02:10:05.288167 (+ 49us) client_negotiation.cc:240] Negotiation successful
0507 02:10:05.288204 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":80,"threads_started":1}
I20240507 02:10:05.288655 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:05.285377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.67:42387 (local address 127.0.0.1:40146)
0507 02:10:05.285572 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:05.285596 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0507 02:10:05.285667 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:10:05.285924 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:10:05.285927 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:10:05.285928 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:10:05.286123 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:10:05.286125 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:10:05.287671 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:10:05.287672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:10:05.288489 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:10:05.288492 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:10:05.288550 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:10:05.288554 (+ 4us) client_negotiation.cc:769] Sending connection context
0507 02:10:05.288581 (+ 27us) client_negotiation.cc:240] Negotiation successful
0507 02:10:05.288615 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":67,"threads_started":1}
I20240507 02:10:05.288949 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:05.285417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.66:38643 (local address 127.0.0.1:38192)
0507 02:10:05.285605 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:05.285613 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0507 02:10:05.285666 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:10:05.286354 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:10:05.286354 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:10:05.286356 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:10:05.286517 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:10:05.286519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:10:05.287792 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:10:05.287794 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:10:05.288757 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:10:05.288760 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:10:05.288819 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:10:05.288822 (+ 3us) client_negotiation.cc:769] Sending connection context
0507 02:10:05.288849 (+ 27us) client_negotiation.cc:240] Negotiation successful
0507 02:10:05.288885 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":86,"threads_started":1}
I20240507 02:10:05.288965 15731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:05.285783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38244 (local address 127.31.131.65:43457)
0507 02:10:05.285938 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0507 02:10:05.285946 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0507 02:10:05.285955 (+ 9us) server_negotiation.cc:407] Connection header received
0507 02:10:05.285994 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:10:05.285996 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:10:05.286010 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:10:05.286104 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:10:05.286326 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:10:05.287192 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:10:05.288119 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:10:05.288774 (+ 655us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:10:05.288824 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0507 02:10:05.288855 (+ 31us) server_negotiation.cc:299] Negotiation successful
0507 02:10:05.288892 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":80,"threads_started":1}
I20240507 02:10:05.289434 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:05.285467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40146 (local address 127.31.131.67:42387)
0507 02:10:05.285792 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0507 02:10:05.285799 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0507 02:10:05.285807 (+ 8us) server_negotiation.cc:407] Connection header received
0507 02:10:05.285834 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:10:05.285836 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:10:05.285847 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:10:05.285911 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:10:05.286214 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:10:05.287595 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:10:05.288810 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:10:05.289263 (+ 453us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:10:05.289307 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0507 02:10:05.289337 (+ 30us) server_negotiation.cc:299] Negotiation successful
0507 02:10:05.289374 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":68,"threads_started":1}
I20240507 02:10:05.289541 15384 tablet_service.cc:1467] Processing CreateTablet for tablet ace17d84b75c4ebe8f2c9d43b75a0087 (DEFAULT_TABLE table=test-table [id=176e92bf5d764c4d8801413348845fef]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:10:05.289738 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:05.285508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38192 (local address 127.31.131.66:38643)
0507 02:10:05.285956 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0507 02:10:05.285963 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0507 02:10:05.285970 (+ 7us) server_negotiation.cc:407] Connection header received
0507 02:10:05.286003 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:10:05.286004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:10:05.286018 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:10:05.286098 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:10:05.286613 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:10:05.287726 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:10:05.289114 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:10:05.289568 (+ 454us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:10:05.289606 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0507 02:10:05.289633 (+ 27us) server_negotiation.cc:299] Negotiation successful
0507 02:10:05.289669 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":76,"threads_started":1}
I20240507 02:10:05.289866 15384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ace17d84b75c4ebe8f2c9d43b75a0087. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:10:05.289894 15652 tablet_service.cc:1467] Processing CreateTablet for tablet ace17d84b75c4ebe8f2c9d43b75a0087 (DEFAULT_TABLE table=test-table [id=176e92bf5d764c4d8801413348845fef]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:10:05.290158 15652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ace17d84b75c4ebe8f2c9d43b75a0087. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:10:05.290210 15518 tablet_service.cc:1467] Processing CreateTablet for tablet ace17d84b75c4ebe8f2c9d43b75a0087 (DEFAULT_TABLE table=test-table [id=176e92bf5d764c4d8801413348845fef]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:10:05.290428 15518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ace17d84b75c4ebe8f2c9d43b75a0087. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:10:05.292289 15732 tablet_bootstrap.cc:492] T ace17d84b75c4ebe8f2c9d43b75a0087 P 0d6e5d3dac474d61a6960acc0b360dbc: Bootstrap starting.
I20240507 02:10:05.292784 15732 tablet_bootstrap.cc:654] T ace17d84b75c4ebe8f2c9d43b75a0087 P 0d6e5d3dac474d61a6960acc0b360dbc: Neither blocks nor log segments found. Creating new log.
I20240507 02:10:05.292995 15733 tablet_bootstrap.cc:492] T ace17d84b75c4ebe8f2c9d43b75a0087 P da19d23f7add4b50bf53330de753ce77: Bootstrap starting.
I20240507 02:10:05.293064 15732 log.cc:824] T ace17d84b75c4ebe8f2c9d43b75a0087 P 0d6e5d3dac474d61a6960acc0b360dbc: Log is configured to *not* fsync() on all Append() calls
I20240507 02:10:05.293242 15734 tablet_bootstrap.cc:492] T ace17d84b75c4ebe8f2c9d43b75a0087 P c5a8b5ba9bf94f4aa922d3d5307406b6: Bootstrap starting.
I20240507 02:10:05.293545 15733 tablet_bootstrap.cc:654] T ace17d84b75c4ebe8f2c9d43b75a0087 P da19d23f7add4b50bf53330de753ce77: Neither blocks nor log segments found. Creating new log.
I20240507 02:10:05.293735 15734 tablet_bootstrap.cc:654] T ace17d84b75c4ebe8f2c9d43b75a0087 P c5a8b5ba9bf94f4aa922d3d5307406b6: Neither blocks nor log segments found. Creating new log.
I20240507 02:10:05.293826 15733 log.cc:824] T ace17d84b75c4ebe8f2c9d43b75a0087 P da19d23f7add4b50bf53330de753ce77: Log is configured to *not* fsync() on all Append() calls
I20240507 02:10:05.293990 15734 log.cc:824] T ace17d84b75c4ebe8f2c9d43b75a0087 P c5a8b5ba9bf94f4aa922d3d5307406b6: Log is configured to *not* fsync() on all Append() calls
I20240507 02:10:05.294502 15733 tablet_bootstrap.cc:492] T ace17d84b75c4ebe8f2c9d43b75a0087 P da19d23f7add4b50bf53330de753ce77: No bootstrap required, opened a new log
I20240507 02:10:05.294574 15733 ts_tablet_manager.cc:1367] T ace17d84b75c4ebe8f2c9d43b75a0087 P da19d23f7add4b50bf53330de753ce77: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240507 02:10:05.294622 15734 tablet_bootstrap.cc:492] T ace17d84b75c4ebe8f2c9d43b75a0087 P c5a8b5ba9bf94f4aa922d3d5307406b6: No bootstrap required, opened a new log
I20240507 02:10:05.294680 15734 ts_tablet_manager.cc:1367] T ace17d84b75c4ebe8f2c9d43b75a0087 P c5a8b5ba9bf94f4aa922d3d5307406b6: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240507 02:10:05.296072 15733 raft_consensus.cc:348] T ace17d84b75c4ebe8f2c9d43b75a0087 P da19d23f7add4b50bf53330de753ce77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da19d23f7add4b50bf53330de753ce77" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 42387 } } peers { permanent_uuid: "0d6e5d3dac474d61a6960acc0b360dbc" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43457 } } peers { permanent_uuid: "c5a8b5ba9bf94f4aa922d3d5307406b6" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 38643 } }
I20240507 02:10:05.296120 15734 raft_consensus.cc:348] T ace17d84b75c4ebe8f2c9d43b75a0087 P c5a8b5ba9bf94f4aa922d3d5307406b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da19d23f7add4b50bf53330de753ce77" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 42387 } } peers { permanent_uuid: "0d6e5d3dac474d61a6960acc0b360dbc" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43457 } } peers { permanent_uuid: "c5a8b5ba9bf94f4aa922d3d5307406b6" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 38643 } }
I20240507 02:10:05.296197 15733 raft_consensus.cc:374] T ace17d84b75c4ebe8f2c9d43b75a0087 P da19d23f7add4b50bf53330de753ce77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:10:05.296221 15733 raft_consensus.cc:729] T ace17d84b75c4ebe8f2c9d43b75a0087 P da19d23f7add4b50bf53330de753ce77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da19d23f7add4b50bf53330de753ce77, State: Initialized, Role: FOLLOWER
I20240507 02:10:05.296216 15734 raft_consensus.cc:374] T ace17d84b75c4ebe8f2c9d43b75a0087 P c5a8b5ba9bf94f4aa922d3d5307406b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:10:05.296238 15734 raft_consensus.cc:729] T ace17d84b75c4ebe8f2c9d43b75a0087 P c5a8b5ba9bf94f4aa922d3d5307406b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5a8b5ba9bf94f4aa922d3d5307406b6, State: Initialized, Role: FOLLOWER
I20240507 02:10:05.296296 15733 consensus_queue.cc:260] T ace17d84b75c4ebe8f2c9d43b75a0087 P da19d23f7add4b50bf53330de753ce77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da19d23f7add4b50bf53330de753ce77" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 42387 } } peers { permanent_uuid: "0d6e5d3dac474d61a6960acc0b360dbc" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43457 } } peers { permanent_uuid: "c5a8b5ba9bf94f4aa922d3d5307406b6" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 38643 } }
I20240507 02:10:05.296319 15734 consensus_queue.cc:260] T ace17d84b75c4ebe8f2c9d43b75a0087 P c5a8b5ba9bf94f4aa922d3d5307406b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da19d23f7add4b50bf53330de753ce77" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 42387 } } peers { permanent_uuid: "0d6e5d3dac474d61a6960acc0b360dbc" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43457 } } peers { permanent_uuid: "c5a8b5ba9bf94f4aa922d3d5307406b6" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 38643 } }
I20240507 02:10:05.296515 15733 ts_tablet_manager.cc:1398] T ace17d84b75c4ebe8f2c9d43b75a0087 P da19d23f7add4b50bf53330de753ce77: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240507 02:10:05.296553 15734 ts_tablet_manager.cc:1398] T ace17d84b75c4ebe8f2c9d43b75a0087 P c5a8b5ba9bf94f4aa922d3d5307406b6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240507 02:10:05.296571 15718 heartbeater.cc:503] Master 127.31.131.126:35343 was elected leader, sending a full tablet report...
I20240507 02:10:05.296643 15584 heartbeater.cc:503] Master 127.31.131.126:35343 was elected leader, sending a full tablet report...
I20240507 02:10:05.296947 15732 tablet_bootstrap.cc:492] T ace17d84b75c4ebe8f2c9d43b75a0087 P 0d6e5d3dac474d61a6960acc0b360dbc: No bootstrap required, opened a new log
I20240507 02:10:05.297015 15732 ts_tablet_manager.cc:1367] T ace17d84b75c4ebe8f2c9d43b75a0087 P 0d6e5d3dac474d61a6960acc0b360dbc: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240507 02:10:05.298818 15732 raft_consensus.cc:348] T ace17d84b75c4ebe8f2c9d43b75a0087 P 0d6e5d3dac474d61a6960acc0b360dbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da19d23f7add4b50bf53330de753ce77" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 42387 } } peers { permanent_uuid: "0d6e5d3dac474d61a6960acc0b360dbc" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43457 } } peers { permanent_uuid: "c5a8b5ba9bf94f4aa922d3d5307406b6" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 38643 } }
I20240507 02:10:05.298924 15732 raft_consensus.cc:374] T ace17d84b75c4ebe8f2c9d43b75a0087 P 0d6e5d3dac474d61a6960acc0b360dbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:10:05.298940 15732 raft_consensus.cc:729] T ace17d84b75c4ebe8f2c9d43b75a0087 P 0d6e5d3dac474d61a6960acc0b360dbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d6e5d3dac474d61a6960acc0b360dbc, State: Initialized, Role: FOLLOWER
I20240507 02:10:05.299000 15732 consensus_queue.cc:260] T ace17d84b75c4ebe8f2c9d43b75a0087 P 0d6e5d3dac474d61a6960acc0b360dbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da19d23f7add4b50bf53330de753ce77" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 42387 } } peers { permanent_uuid: "0d6e5d3dac474d61a6960acc0b360dbc" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43457 } } peers { permanent_uuid: "c5a8b5ba9bf94f4aa922d3d5307406b6" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 38643 } }
I20240507 02:10:05.299170 15732 ts_tablet_manager.cc:1398] T ace17d84b75c4ebe8f2c9d43b75a0087 P 0d6e5d3dac474d61a6960acc0b360dbc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240507 02:10:05.299270 15450 heartbeater.cc:503] Master 127.31.131.126:35343 was elected leader, sending a full tablet report...
I20240507 02:10:05.351024 15738 raft_consensus.cc:482] T ace17d84b75c4ebe8f2c9d43b75a0087 P da19d23f7add4b50bf53330de753ce77 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 02:10:05.351156 15738 raft_consensus.cc:504] T ace17d84b75c4ebe8f2c9d43b75a0087 P da19d23f7add4b50bf53330de753ce77 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da19d23f7add4b50bf53330de753ce77" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 42387 } } peers { permanent_uuid: "0d6e5d3dac474d61a6960acc0b360dbc" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43457 } } peers { permanent_uuid: "c5a8b5ba9bf94f4aa922d3d5307406b6" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 38643 } }
I20240507 02:10:05.351470 15738 leader_election.cc:288] T ace17d84b75c4ebe8f2c9d43b75a0087 P da19d23f7add4b50bf53330de753ce77 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d6e5d3dac474d61a6960acc0b360dbc (127.31.131.65:43457), c5a8b5ba9bf94f4aa922d3d5307406b6 (127.31.131.66:38643)
I20240507 02:10:05.354230 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:05.351636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.65:43457 (local address 127.31.131.67:55663)
0507 02:10:05.351700 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:05.351712 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0507 02:10:05.351752 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:10:05.351902 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:10:05.351903 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:10:05.351905 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:10:05.352127 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:10:05.352129 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:10:05.353022 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:10:05.353023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:10:05.353963 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:10:05.353965 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:10:05.354026 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:10:05.354028 (+ 2us) client_negotiation.cc:769] Sending connection context
0507 02:10:05.354058 (+ 30us) client_negotiation.cc:240] Negotiation successful
0507 02:10:05.354089 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20240507 02:10:05.354238 15741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:05.351636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.66:38643 (local address 127.31.131.67:45073)
0507 02:10:05.351811 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:05.351819 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0507 02:10:05.351867 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:10:05.351985 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:10:05.351986 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:10:05.351987 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:10:05.352203 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:10:05.352205 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:10:05.353031 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:10:05.353032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:10:05.353950 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:10:05.353953 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:10:05.354024 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:10:05.354028 (+ 4us) client_negotiation.cc:769] Sending connection context
0507 02:10:05.354058 (+ 30us) client_negotiation.cc:240] Negotiation successful
0507 02:10:05.354089 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":12,"thread_start_us":85,"threads_started":1}
I20240507 02:10:05.354661 15731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:05.351708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.67:55663 (local address 127.31.131.65:43457)
0507 02:10:05.351755 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0507 02:10:05.351756 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:10:05.351767 (+ 11us) server_negotiation.cc:407] Connection header received
0507 02:10:05.351820 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:10:05.351821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:10:05.351828 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:10:05.351891 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:10:05.352200 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:10:05.352945 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:10:05.354028 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:10:05.354509 (+ 481us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:10:05.354516 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0507 02:10:05.354538 (+ 22us) server_negotiation.cc:299] Negotiation successful
0507 02:10:05.354599 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20240507 02:10:05.354666 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:05.351733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.67:45073 (local address 127.31.131.66:38643)
0507 02:10:05.351772 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0507 02:10:05.351774 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:10:05.351877 (+ 103us) server_negotiation.cc:407] Connection header received
0507 02:10:05.351918 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:10:05.351919 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:10:05.351928 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:10:05.351968 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:10:05.352264 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:10:05.352969 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:10:05.354028 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:10:05.354508 (+ 480us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:10:05.354516 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0507 02:10:05.354538 (+ 22us) server_negotiation.cc:299] Negotiation successful
0507 02:10:05.354595 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20240507 02:10:05.354931 15404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ace17d84b75c4ebe8f2c9d43b75a0087" candidate_uuid: "da19d23f7add4b50bf53330de753ce77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d6e5d3dac474d61a6960acc0b360dbc" is_pre_election: true
I20240507 02:10:05.355065 15404 raft_consensus.cc:2454] T ace17d84b75c4ebe8f2c9d43b75a0087 P 0d6e5d3dac474d61a6960acc0b360dbc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da19d23f7add4b50bf53330de753ce77 in term 0.
I20240507 02:10:05.355034 15538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ace17d84b75c4ebe8f2c9d43b75a0087" candidate_uuid: "da19d23f7add4b50bf53330de753ce77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5a8b5ba9bf94f4aa922d3d5307406b6" is_pre_election: true
I20240507 02:10:05.355170 15538 raft_consensus.cc:2454] T ace17d84b75c4ebe8f2c9d43b75a0087 P c5a8b5ba9bf94f4aa922d3d5307406b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da19d23f7add4b50bf53330de753ce77 in term 0.
I20240507 02:10:05.355262 15607 leader_election.cc:302] T ace17d84b75c4ebe8f2c9d43b75a0087 P da19d23f7add4b50bf53330de753ce77 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d6e5d3dac474d61a6960acc0b360dbc, da19d23f7add4b50bf53330de753ce77; no voters:
I20240507 02:10:05.355397 15738 raft_consensus.cc:2781] T ace17d84b75c4ebe8f2c9d43b75a0087 P da19d23f7add4b50bf53330de753ce77 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 02:10:05.355463 15738 raft_consensus.cc:482] T ace17d84b75c4ebe8f2c9d43b75a0087 P da19d23f7add4b50bf53330de753ce77 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 02:10:05.355489 15738 raft_consensus.cc:3037] T ace17d84b75c4ebe8f2c9d43b75a0087 P da19d23f7add4b50bf53330de753ce77 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:10:05.356151 15738 raft_consensus.cc:504] T ace17d84b75c4ebe8f2c9d43b75a0087 P da19d23f7add4b50bf53330de753ce77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da19d23f7add4b50bf53330de753ce77" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 42387 } } peers { permanent_uuid: "0d6e5d3dac474d61a6960acc0b360dbc" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43457 } } peers { permanent_uuid: "c5a8b5ba9bf94f4aa922d3d5307406b6" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 38643 } }
I20240507 02:10:05.356276 15738 leader_election.cc:288] T ace17d84b75c4ebe8f2c9d43b75a0087 P da19d23f7add4b50bf53330de753ce77 [CANDIDATE]: Term 1 election: Requested vote from peers 0d6e5d3dac474d61a6960acc0b360dbc (127.31.131.65:43457), c5a8b5ba9bf94f4aa922d3d5307406b6 (127.31.131.66:38643)
I20240507 02:10:05.356428 15404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ace17d84b75c4ebe8f2c9d43b75a0087" candidate_uuid: "da19d23f7add4b50bf53330de753ce77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d6e5d3dac474d61a6960acc0b360dbc"
I20240507 02:10:05.356493 15404 raft_consensus.cc:3037] T ace17d84b75c4ebe8f2c9d43b75a0087 P 0d6e5d3dac474d61a6960acc0b360dbc [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:10:05.357151 15404 raft_consensus.cc:2454] T ace17d84b75c4ebe8f2c9d43b75a0087 P 0d6e5d3dac474d61a6960acc0b360dbc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da19d23f7add4b50bf53330de753ce77 in term 1.
I20240507 02:10:05.357255 15538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ace17d84b75c4ebe8f2c9d43b75a0087" candidate_uuid: "da19d23f7add4b50bf53330de753ce77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5a8b5ba9bf94f4aa922d3d5307406b6"
I20240507 02:10:05.357321 15607 leader_election.cc:302] T ace17d84b75c4ebe8f2c9d43b75a0087 P da19d23f7add4b50bf53330de753ce77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d6e5d3dac474d61a6960acc0b360dbc, da19d23f7add4b50bf53330de753ce77; no voters:
I20240507 02:10:05.357333 15538 raft_consensus.cc:3037] T ace17d84b75c4ebe8f2c9d43b75a0087 P c5a8b5ba9bf94f4aa922d3d5307406b6 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:10:05.357436 15738 raft_consensus.cc:2781] T ace17d84b75c4ebe8f2c9d43b75a0087 P da19d23f7add4b50bf53330de753ce77 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:10:05.357580 15738 raft_consensus.cc:686] T ace17d84b75c4ebe8f2c9d43b75a0087 P da19d23f7add4b50bf53330de753ce77 [term 1 LEADER]: Becoming Leader. State: Replica: da19d23f7add4b50bf53330de753ce77, State: Running, Role: LEADER
I20240507 02:10:05.357667 15738 consensus_queue.cc:237] T ace17d84b75c4ebe8f2c9d43b75a0087 P da19d23f7add4b50bf53330de753ce77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da19d23f7add4b50bf53330de753ce77" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 42387 } } peers { permanent_uuid: "0d6e5d3dac474d61a6960acc0b360dbc" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43457 } } peers { permanent_uuid: "c5a8b5ba9bf94f4aa922d3d5307406b6" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 38643 } }
I20240507 02:10:05.358170 15538 raft_consensus.cc:2454] T ace17d84b75c4ebe8f2c9d43b75a0087 P c5a8b5ba9bf94f4aa922d3d5307406b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da19d23f7add4b50bf53330de753ce77 in term 1.
I20240507 02:10:05.358454 15259 catalog_manager.cc:5484] T ace17d84b75c4ebe8f2c9d43b75a0087 P da19d23f7add4b50bf53330de753ce77 reported cstate change: term changed from 0 to 1, leader changed from <none> to da19d23f7add4b50bf53330de753ce77 (127.31.131.67). New cstate: current_term: 1 leader_uuid: "da19d23f7add4b50bf53330de753ce77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da19d23f7add4b50bf53330de753ce77" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 42387 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d6e5d3dac474d61a6960acc0b360dbc" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5a8b5ba9bf94f4aa922d3d5307406b6" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 38643 } health_report { overall_health: UNKNOWN } } }
W20240507 02:10:05.366704 15585 tablet.cc:2369] T ace17d84b75c4ebe8f2c9d43b75a0087 P c5a8b5ba9bf94f4aa922d3d5307406b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240507 02:10:05.379841 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:05.378025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40152 (local address 127.31.131.67:42387)
0507 02:10:05.378076 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0507 02:10:05.378077 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:10:05.378085 (+ 8us) server_negotiation.cc:407] Connection header received
0507 02:10:05.378122 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:10:05.378123 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:10:05.378139 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:10:05.378200 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0507 02:10:05.378397 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:10:05.378913 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:10:05.379423 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:10:05.379530 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:10:05.379562 (+ 32us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0507 02:10:05.379651 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0507 02:10:05.379706 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0507 02:10:05.379750 (+ 44us) server_negotiation.cc:299] Negotiation successful
0507 02:10:05.379767 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20240507 02:10:05.381319 15538 raft_consensus.cc:1261] T ace17d84b75c4ebe8f2c9d43b75a0087 P c5a8b5ba9bf94f4aa922d3d5307406b6 [term 1 FOLLOWER]: Refusing update from remote peer da19d23f7add4b50bf53330de753ce77: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 02:10:05.381326 15404 raft_consensus.cc:1261] T ace17d84b75c4ebe8f2c9d43b75a0087 P 0d6e5d3dac474d61a6960acc0b360dbc [term 1 FOLLOWER]: Refusing update from remote peer da19d23f7add4b50bf53330de753ce77: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 02:10:05.381615 15742 consensus_queue.cc:1035] T ace17d84b75c4ebe8f2c9d43b75a0087 P da19d23f7add4b50bf53330de753ce77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5a8b5ba9bf94f4aa922d3d5307406b6" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 38643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 02:10:05.381731 15738 consensus_queue.cc:1035] T ace17d84b75c4ebe8f2c9d43b75a0087 P da19d23f7add4b50bf53330de753ce77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d6e5d3dac474d61a6960acc0b360dbc" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 02:10:05.384035 15747 mvcc.cc:205] Tried to move back new op lower bound from 7024835810839740416 to 7024835810745585664. Current Snapshot: MvccSnapshot[applied={T|T < 7024835810839740416}]
I20240507 02:10:05.384608 15746 mvcc.cc:205] Tried to move back new op lower bound from 7024835810839740416 to 7024835810745585664. Current Snapshot: MvccSnapshot[applied={T|T < 7024835810839740416}]
I20240507 02:10:05.385649 15259 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47390:
table { table_name: "test-table" } modify_external_catalogs: true
I20240507 02:10:05.385730 15259 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47390:
table { table_name: "test-table" } modify_external_catalogs: true
I20240507 02:10:05.386382 15259 catalog_manager.cc:5771] T 00000000000000000000000000000000 P dec8ad1f575d48a59f9845b310f4af5b: Sending DeleteTablet for 3 replicas of tablet ace17d84b75c4ebe8f2c9d43b75a0087
I20240507 02:10:05.386668 15652 tablet_service.cc:1514] Processing DeleteTablet for tablet ace17d84b75c4ebe8f2c9d43b75a0087 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:10:05 UTC) from {username='kudu', principal='kudu/127.31.131.126@KRBTEST.COM'} at 127.0.0.1:40146
I20240507 02:10:05.386723 15384 tablet_service.cc:1514] Processing DeleteTablet for tablet ace17d84b75c4ebe8f2c9d43b75a0087 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:10:05 UTC) from {username='kudu', principal='kudu/127.31.131.126@KRBTEST.COM'} at 127.0.0.1:38244
I20240507 02:10:05.386716 15518 tablet_service.cc:1514] Processing DeleteTablet for tablet ace17d84b75c4ebe8f2c9d43b75a0087 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:10:05 UTC) from {username='kudu', principal='kudu/127.31.131.126@KRBTEST.COM'} at 127.0.0.1:38192
I20240507 02:10:05.386901 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 15321
I20240507 02:10:05.389178 15757 tablet_replica.cc:330] T ace17d84b75c4ebe8f2c9d43b75a0087 P c5a8b5ba9bf94f4aa922d3d5307406b6: stopping tablet replica
I20240507 02:10:05.389251 15757 raft_consensus.cc:2229] T ace17d84b75c4ebe8f2c9d43b75a0087 P c5a8b5ba9bf94f4aa922d3d5307406b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 02:10:05.389338 15757 raft_consensus.cc:2258] T ace17d84b75c4ebe8f2c9d43b75a0087 P c5a8b5ba9bf94f4aa922d3d5307406b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 02:10:05.389477 15755 tablet_replica.cc:330] T ace17d84b75c4ebe8f2c9d43b75a0087 P da19d23f7add4b50bf53330de753ce77: stopping tablet replica
I20240507 02:10:05.389549 15755 raft_consensus.cc:2229] T ace17d84b75c4ebe8f2c9d43b75a0087 P da19d23f7add4b50bf53330de753ce77 [term 1 LEADER]: Raft consensus shutting down.
I20240507 02:10:05.389652 15755 raft_consensus.cc:2258] T ace17d84b75c4ebe8f2c9d43b75a0087 P da19d23f7add4b50bf53330de753ce77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 02:10:05.389971 15755 ts_tablet_manager.cc:1875] T ace17d84b75c4ebe8f2c9d43b75a0087 P da19d23f7add4b50bf53330de753ce77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 02:10:05.389968 15757 ts_tablet_manager.cc:1875] T ace17d84b75c4ebe8f2c9d43b75a0087 P c5a8b5ba9bf94f4aa922d3d5307406b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 02:10:05.391470 15755 ts_tablet_manager.cc:1888] T ace17d84b75c4ebe8f2c9d43b75a0087 P da19d23f7add4b50bf53330de753ce77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240507 02:10:05.391536 15757 ts_tablet_manager.cc:1888] T ace17d84b75c4ebe8f2c9d43b75a0087 P c5a8b5ba9bf94f4aa922d3d5307406b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240507 02:10:05.391605 15757 log.cc:1196] T ace17d84b75c4ebe8f2c9d43b75a0087 P c5a8b5ba9bf94f4aa922d3d5307406b6: Deleting WAL directory at /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-1/wal/wals/ace17d84b75c4ebe8f2c9d43b75a0087
I20240507 02:10:05.391625 15755 log.cc:1196] T ace17d84b75c4ebe8f2c9d43b75a0087 P da19d23f7add4b50bf53330de753ce77: Deleting WAL directory at /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-2/wal/wals/ace17d84b75c4ebe8f2c9d43b75a0087
I20240507 02:10:05.391876 15757 ts_tablet_manager.cc:1909] T ace17d84b75c4ebe8f2c9d43b75a0087 P c5a8b5ba9bf94f4aa922d3d5307406b6: Deleting consensus metadata
I20240507 02:10:05.392073 15755 ts_tablet_manager.cc:1909] T ace17d84b75c4ebe8f2c9d43b75a0087 P da19d23f7add4b50bf53330de753ce77: Deleting consensus metadata
I20240507 02:10:05.392362 15246 catalog_manager.cc:4830] TS c5a8b5ba9bf94f4aa922d3d5307406b6 (127.31.131.66:38643): tablet ace17d84b75c4ebe8f2c9d43b75a0087 (table test-table [id=176e92bf5d764c4d8801413348845fef]) successfully deleted
I20240507 02:10:05.392418 15245 catalog_manager.cc:4830] TS da19d23f7add4b50bf53330de753ce77 (127.31.131.67:42387): tablet ace17d84b75c4ebe8f2c9d43b75a0087 (table test-table [id=176e92bf5d764c4d8801413348845fef]) successfully deleted
W20240507 02:10:05.393345 15245 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.131.65:43457 (error 108)
I20240507 02:10:05.393563 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 15455
I20240507 02:10:05.394260 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:05.393698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.65:43457 (local address 127.0.0.1:38256)
0507 02:10:05.394193 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:05.394227 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.65:43457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":474}
W20240507 02:10:05.395210 15245 catalog_manager.cc:4573] TS 0d6e5d3dac474d61a6960acc0b360dbc (127.31.131.65:43457): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ace17d84b75c4ebe8f2c9d43b75a0087: Network error: Client connection negotiation failed: client connection to 127.31.131.65:43457: connect: Connection refused (error 111)
I20240507 02:10:05.399179 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 15589
I20240507 02:10:05.403249 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 15227
I20240507 02:10:25.407677 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:35343
--webserver_interface=127.31.131.126
--webserver_port=41271
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab
--principal=kudu/127.31.131.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:43653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.131.126:35343
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:10:25.483780 15759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:10:25.483969 15759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:10:25.483997 15759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:10:25.485275 15759 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20240507 02:10:25.485312 15759 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20240507 02:10:25.485329 15759 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20240507 02:10:25.485342 15759 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240507 02:10:25.485357 15759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:10:25.485373 15759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:10:25.485399 15759 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240507 02:10:25.485415 15759 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240507 02:10:25.487048 15759 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:43653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.31.131.126:35343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab
--principal=kudu/127.31.131.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:35343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.131.126
--webserver_port=41271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:10:25.487253 15759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:10:25.487418 15759 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:10:25.493810 15766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:10:25.493813 15768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:10:25.493834 15759 server_base.cc:1034] running on GCE node
W20240507 02:10:25.493810 15765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:10:25.494248 15759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:10:25.494494 15759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:10:25.495646 15759 hybrid_clock.cc:648] HybridClock initialized: now 1715047825495634 us; error 38 us; skew 500 ppm
May 07 02:10:25 dist-test-slave-v0bg krb5kdc[15214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047825, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:10:25.503147 15759 init.cc:377] Logged in from keytab as kudu/127.31.131.126@KRBTEST.COM (short username kudu)
I20240507 02:10:25.503535 15759 webserver.cc:460] Webserver started at http://127.31.131.126:41271/ using document root <none> and password file <none>
I20240507 02:10:25.503710 15759 fs_manager.cc:359] Metadata directory not provided
I20240507 02:10:25.503763 15759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:10:25.504967 15759 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20240507 02:10:25.505491 15775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:10:25.505705 15759 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240507 02:10:25.505790 15759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/master-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/master-0/wal
uuid: "dec8ad1f575d48a59f9845b310f4af5b"
format_stamp: "Formatted at 2024-05-07 02:10:04 on dist-test-slave-v0bg"
server_key: "0b9180c36ad370f362912ab9d5c00dbc"
server_key_iv: "e25f55c88aa80c06a136d70722054126"
server_key_version: "encryptionkey@0"
I20240507 02:10:25.506129 15759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:10:25.552531 15759 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:10:25.554438 15759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:10:25.554581 15759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:10:25.558205 15759 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.126:35343
I20240507 02:10:25.558248 15827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.126:35343 every 8 connection(s)
I20240507 02:10:25.558580 15759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
I20240507 02:10:25.559823 15828 sys_catalog.cc:263] Verifying existing consensus state
I20240507 02:10:25.560333 15828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dec8ad1f575d48a59f9845b310f4af5b: Bootstrap starting.
I20240507 02:10:25.562150 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 15759
I20240507 02:10:25.562386 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.65:43457
--local_ip_for_outbound_sockets=127.31.131.65
--tserver_master_addrs=127.31.131.126:35343
--webserver_port=40345
--webserver_interface=127.31.131.65
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab
--principal=kudu/127.31.131.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:43653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240507 02:10:25.569108 15828 log.cc:824] T 00000000000000000000000000000000 P dec8ad1f575d48a59f9845b310f4af5b: Log is configured to *not* fsync() on all Append() calls
I20240507 02:10:25.570909 15828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dec8ad1f575d48a59f9845b310f4af5b: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240507 02:10:25.571148 15828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dec8ad1f575d48a59f9845b310f4af5b: Bootstrap complete.
I20240507 02:10:25.573071 15828 raft_consensus.cc:348] T 00000000000000000000000000000000 P dec8ad1f575d48a59f9845b310f4af5b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec8ad1f575d48a59f9845b310f4af5b" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 35343 } }
I20240507 02:10:25.573262 15828 raft_consensus.cc:729] T 00000000000000000000000000000000 P dec8ad1f575d48a59f9845b310f4af5b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec8ad1f575d48a59f9845b310f4af5b, State: Initialized, Role: FOLLOWER
I20240507 02:10:25.573381 15828 consensus_queue.cc:260] T 00000000000000000000000000000000 P dec8ad1f575d48a59f9845b310f4af5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec8ad1f575d48a59f9845b310f4af5b" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 35343 } }
I20240507 02:10:25.573439 15828 raft_consensus.cc:388] T 00000000000000000000000000000000 P dec8ad1f575d48a59f9845b310f4af5b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 02:10:25.573475 15828 raft_consensus.cc:482] T 00000000000000000000000000000000 P dec8ad1f575d48a59f9845b310f4af5b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 02:10:25.573522 15828 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dec8ad1f575d48a59f9845b310f4af5b [term 1 FOLLOWER]: Advancing to term 2
I20240507 02:10:25.574375 15828 raft_consensus.cc:504] T 00000000000000000000000000000000 P dec8ad1f575d48a59f9845b310f4af5b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec8ad1f575d48a59f9845b310f4af5b" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 35343 } }
I20240507 02:10:25.574456 15828 leader_election.cc:302] T 00000000000000000000000000000000 P dec8ad1f575d48a59f9845b310f4af5b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dec8ad1f575d48a59f9845b310f4af5b; no voters:
I20240507 02:10:25.574628 15828 leader_election.cc:288] T 00000000000000000000000000000000 P dec8ad1f575d48a59f9845b310f4af5b [CANDIDATE]: Term 2 election: Requested vote from peers
I20240507 02:10:25.574664 15832 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dec8ad1f575d48a59f9845b310f4af5b [term 2 FOLLOWER]: Leader election won for term 2
I20240507 02:10:25.574815 15832 raft_consensus.cc:686] T 00000000000000000000000000000000 P dec8ad1f575d48a59f9845b310f4af5b [term 2 LEADER]: Becoming Leader. State: Replica: dec8ad1f575d48a59f9845b310f4af5b, State: Running, Role: LEADER
I20240507 02:10:25.574864 15828 sys_catalog.cc:564] T 00000000000000000000000000000000 P dec8ad1f575d48a59f9845b310f4af5b [sys.catalog]: configured and running, proceeding with master startup.
I20240507 02:10:25.574967 15832 consensus_queue.cc:237] T 00000000000000000000000000000000 P dec8ad1f575d48a59f9845b310f4af5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec8ad1f575d48a59f9845b310f4af5b" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 35343 } }
I20240507 02:10:25.575285 15833 sys_catalog.cc:455] T 00000000000000000000000000000000 P dec8ad1f575d48a59f9845b310f4af5b [sys.catalog]: SysCatalogTable state changed. Reason: New leader dec8ad1f575d48a59f9845b310f4af5b. Latest consensus state: current_term: 2 leader_uuid: "dec8ad1f575d48a59f9845b310f4af5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec8ad1f575d48a59f9845b310f4af5b" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 35343 } } }
I20240507 02:10:25.575279 15834 sys_catalog.cc:455] T 00000000000000000000000000000000 P dec8ad1f575d48a59f9845b310f4af5b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "dec8ad1f575d48a59f9845b310f4af5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec8ad1f575d48a59f9845b310f4af5b" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 35343 } } }
I20240507 02:10:25.575358 15833 sys_catalog.cc:458] T 00000000000000000000000000000000 P dec8ad1f575d48a59f9845b310f4af5b [sys.catalog]: This master's current role is: LEADER
I20240507 02:10:25.575366 15834 sys_catalog.cc:458] T 00000000000000000000000000000000 P dec8ad1f575d48a59f9845b310f4af5b [sys.catalog]: This master's current role is: LEADER
I20240507 02:10:25.575922 15837 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 02:10:25.576495 15837 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240507 02:10:25.576581 15837 catalog_manager.cc:1259] Loaded cluster ID: 69cbc585993f46de82c1ce606effdf61
I20240507 02:10:25.576620 15837 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240507 02:10:25.577499 15837 catalog_manager.cc:1504] Loading token signing keys...
I20240507 02:10:25.577663 15837 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dec8ad1f575d48a59f9845b310f4af5b: Loaded TSK: 0
I20240507 02:10:25.583169 15837 catalog_manager.cc:5857] T 00000000000000000000000000000000 P dec8ad1f575d48a59f9845b310f4af5b: Generated new TSK 1
I20240507 02:10:25.583257 15837 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20240507 02:10:25.645493 15830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:10:25.645673 15830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:10:25.645694 15830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:10:25.646957 15830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:10:25.646996 15830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:10:25.647018 15830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.65
I20240507 02:10:25.648509 15830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:43653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab
--principal=kudu/127.31.131.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.65:43457
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.131.65
--webserver_port=40345
--tserver_master_addrs=127.31.131.126:35343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.65
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:10:25.648701 15830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:10:25.648873 15830 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:10:25.652599 15856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:10:25.652830 15830 server_base.cc:1034] running on GCE node
W20240507 02:10:25.652837 15858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:10:25.652599 15855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:10:25.653103 15830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:10:25.653297 15830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:10:25.654421 15830 hybrid_clock.cc:648] HybridClock initialized: now 1715047825654401 us; error 35 us; skew 500 ppm
May 07 02:10:25 dist-test-slave-v0bg krb5kdc[15214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047825, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:10:25.656749 15830 init.cc:377] Logged in from keytab as kudu/127.31.131.65@KRBTEST.COM (short username kudu)
I20240507 02:10:25.657145 15830 webserver.cc:460] Webserver started at http://127.31.131.65:40345/ using document root <none> and password file <none>
I20240507 02:10:25.657303 15830 fs_manager.cc:359] Metadata directory not provided
I20240507 02:10:25.657353 15830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:10:25.658632 15830 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20240507 02:10:25.659219 15865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:10:25.659405 15830 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240507 02:10:25.659482 15830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-0/wal
uuid: "0d6e5d3dac474d61a6960acc0b360dbc"
format_stamp: "Formatted at 2024-05-07 02:10:04 on dist-test-slave-v0bg"
server_key: "5eeb8e1ff74dba96a2a309487f9cf3ea"
server_key_iv: "33df10cbc886a1a47eeff61fa4082c42"
server_key_version: "encryptionkey@0"
I20240507 02:10:25.659760 15830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:10:25.673097 15830 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:10:25.675097 15830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:10:25.675240 15830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:10:25.675451 15830 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:10:25.675913 15872 ts_tablet_manager.cc:516] Loading tablet metadata (0/1 complete)
I20240507 02:10:25.676836 15830 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240507 02:10:25.676887 15830 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20240507 02:10:25.676910 15830 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240507 02:10:25.677429 15830 ts_tablet_manager.cc:590] Registered 1 tablets
I20240507 02:10:25.677466 15830 ts_tablet_manager.cc:569] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20240507 02:10:25.677517 15872 tablet_bootstrap.cc:492] T ace17d84b75c4ebe8f2c9d43b75a0087 P 0d6e5d3dac474d61a6960acc0b360dbc: Bootstrap starting.
I20240507 02:10:25.684689 15830 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.65:43457
I20240507 02:10:25.684767 15979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.65:43457 every 8 connection(s)
I20240507 02:10:25.685194 15830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
I20240507 02:10:25.686388 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 15830
I20240507 02:10:25.686748 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.66:38643
--local_ip_for_outbound_sockets=127.31.131.66
--tserver_master_addrs=127.31.131.126:35343
--webserver_port=32811
--webserver_interface=127.31.131.66
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab
--principal=kudu/127.31.131.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:43653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240507 02:10:25.688724 15872 log.cc:824] T ace17d84b75c4ebe8f2c9d43b75a0087 P 0d6e5d3dac474d61a6960acc0b360dbc: Log is configured to *not* fsync() on all Append() calls
I20240507 02:10:25.691264 15872 tablet_bootstrap.cc:492] T ace17d84b75c4ebe8f2c9d43b75a0087 P 0d6e5d3dac474d61a6960acc0b360dbc: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240507 02:10:25.691615 15872 tablet_bootstrap.cc:492] T ace17d84b75c4ebe8f2c9d43b75a0087 P 0d6e5d3dac474d61a6960acc0b360dbc: Bootstrap complete.
I20240507 02:10:25.692735 15872 ts_tablet_manager.cc:1367] T ace17d84b75c4ebe8f2c9d43b75a0087 P 0d6e5d3dac474d61a6960acc0b360dbc: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.004s
I20240507 02:10:25.693962 15872 raft_consensus.cc:348] T ace17d84b75c4ebe8f2c9d43b75a0087 P 0d6e5d3dac474d61a6960acc0b360dbc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da19d23f7add4b50bf53330de753ce77" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 42387 } } peers { permanent_uuid: "0d6e5d3dac474d61a6960acc0b360dbc" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43457 } } peers { permanent_uuid: "c5a8b5ba9bf94f4aa922d3d5307406b6" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 38643 } }
I20240507 02:10:25.694373 15872 raft_consensus.cc:729] T ace17d84b75c4ebe8f2c9d43b75a0087 P 0d6e5d3dac474d61a6960acc0b360dbc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d6e5d3dac474d61a6960acc0b360dbc, State: Initialized, Role: FOLLOWER
I20240507 02:10:25.694537 15872 consensus_queue.cc:260] T ace17d84b75c4ebe8f2c9d43b75a0087 P 0d6e5d3dac474d61a6960acc0b360dbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da19d23f7add4b50bf53330de753ce77" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 42387 } } peers { permanent_uuid: "0d6e5d3dac474d61a6960acc0b360dbc" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43457 } } peers { permanent_uuid: "c5a8b5ba9bf94f4aa922d3d5307406b6" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 38643 } }
I20240507 02:10:25.694873 15872 ts_tablet_manager.cc:1398] T ace17d84b75c4ebe8f2c9d43b75a0087 P 0d6e5d3dac474d61a6960acc0b360dbc: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
May 07 02:10:25 dist-test-slave-v0bg krb5kdc[15214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047825, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.65@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:10:25.701444 15982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:25.685963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:35343 (local address 127.31.131.65:51121)
0507 02:10:25.690816 (+ 4853us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:25.690851 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0507 02:10:25.691433 (+ 582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:10:25.692211 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:10:25.692215 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:10:25.692612 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:10:25.692998 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:10:25.693004 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:10:25.694523 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:10:25.694524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:10:25.695157 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:10:25.695160 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:10:25.695287 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:10:25.695893 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:10:25.695914 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:10:25.697668 (+ 1754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:10:25.700003 (+ 2335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:10:25.700009 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:10:25.700012 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:10:25.700263 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:10:25.700487 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:10:25.700487 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:10:25.700488 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:10:25.700558 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:10:25.700907 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:10:25.700912 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:10:25.701038 (+ 126us) client_negotiation.cc:769] Sending connection context
0507 02:10:25.701155 (+ 117us) client_negotiation.cc:240] Negotiation successful
0507 02:10:25.701281 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4811,"thread_start_us":84,"threads_started":1}
I20240507 02:10:25.701675 15983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:25.686052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:51121 (local address 127.31.131.126:35343)
0507 02:10:25.691642 (+ 5590us) server_negotiation.cc:206] Beginning negotiation
0507 02:10:25.691653 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0507 02:10:25.691678 (+ 25us) server_negotiation.cc:407] Connection header received
0507 02:10:25.691806 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:10:25.691811 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:10:25.691858 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:10:25.691990 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:10:25.693151 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:10:25.694409 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:10:25.695254 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:10:25.695876 (+ 622us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:10:25.697781 (+ 1905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:10:25.697804 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:10:25.697805 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:10:25.697824 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:10:25.699915 (+ 2091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:10:25.700345 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:10:25.700348 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:10:25.700349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:10:25.700412 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:10:25.700620 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:10:25.700620 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:10:25.700621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:10:25.700838 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:10:25.700934 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0507 02:10:25.701223 (+ 289us) server_negotiation.cc:299] Negotiation successful
0507 02:10:25.701351 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5544,"thread_start_us":96,"threads_started":1}
I20240507 02:10:25.702458 15980 heartbeater.cc:347] Connected to a master server at 127.31.131.126:35343
I20240507 02:10:25.702577 15980 heartbeater.cc:464] Registering TS with master...
I20240507 02:10:25.702775 15980 heartbeater.cc:511] Master 127.31.131.126:35343 requested a full tablet report, sending...
I20240507 02:10:25.703482 15792 ts_manager.cc:194] Registered new tserver with Master: 0d6e5d3dac474d61a6960acc0b360dbc (127.31.131.65:43457)
I20240507 02:10:25.704717 15792 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.131.65@KRBTEST.COM'} at 127.31.131.65:51121
I20240507 02:10:25.705600 15980 heartbeater.cc:503] Master 127.31.131.126:35343 was elected leader, sending a full tablet report...
May 07 02:10:25 dist-test-slave-v0bg krb5kdc[15214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047825, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.126@KRBTEST.COM for kudu/127.31.131.65@KRBTEST.COM
I20240507 02:10:25.713287 15987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:25.704079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.65:43457 (local address 127.0.0.1:60910)
0507 02:10:25.704251 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:25.704276 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0507 02:10:25.704342 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:10:25.705279 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:10:25.705285 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:10:25.705624 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:10:25.706145 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:10:25.706149 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:10:25.707460 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:10:25.707462 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:10:25.708055 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:10:25.708057 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:10:25.708117 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:10:25.708618 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:10:25.708622 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:10:25.710293 (+ 1671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:10:25.712277 (+ 1984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:10:25.712282 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:10:25.712284 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:10:25.712488 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:10:25.712679 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:10:25.712680 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:10:25.712681 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:10:25.712711 (+ 30us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:10:25.712936 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:10:25.712940 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:10:25.712985 (+ 45us) client_negotiation.cc:769] Sending connection context
0507 02:10:25.713063 (+ 78us) client_negotiation.cc:240] Negotiation successful
0507 02:10:25.713163 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":98,"threads_started":1}
I20240507 02:10:25.713647 15988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:25.704960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60910 (local address 127.31.131.65:43457)
0507 02:10:25.705121 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0507 02:10:25.705129 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0507 02:10:25.705139 (+ 10us) server_negotiation.cc:407] Connection header received
0507 02:10:25.705168 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:10:25.705170 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:10:25.705184 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:10:25.706227 (+ 1043us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:10:25.706285 (+ 58us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:10:25.707347 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:10:25.708334 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:10:25.708562 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:10:25.710403 (+ 1841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:10:25.710410 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:10:25.710410 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:10:25.710415 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:10:25.712180 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:10:25.712578 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:10:25.712581 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:10:25.712581 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:10:25.712613 (+ 32us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:10:25.712767 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:10:25.712768 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:10:25.712768 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:10:25.712863 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:10:25.713432 (+ 569us) server_negotiation.cc:1035] Waiting for connection context
0507 02:10:25.713480 (+ 48us) server_negotiation.cc:299] Negotiation successful
0507 02:10:25.713562 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":87,"threads_started":1}
I20240507 02:10:25.714097 15914 tablet_service.cc:1514] Processing DeleteTablet for tablet ace17d84b75c4ebe8f2c9d43b75a0087 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:10:05 UTC) from {username='kudu', principal='kudu/127.31.131.126@KRBTEST.COM'} at 127.0.0.1:60910
I20240507 02:10:25.714715 15989 tablet_replica.cc:330] stopping tablet replica
I20240507 02:10:25.714810 15989 raft_consensus.cc:2229] T ace17d84b75c4ebe8f2c9d43b75a0087 P 0d6e5d3dac474d61a6960acc0b360dbc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 02:10:25.714905 15989 raft_consensus.cc:2258] T ace17d84b75c4ebe8f2c9d43b75a0087 P 0d6e5d3dac474d61a6960acc0b360dbc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 02:10:25.715245 15989 ts_tablet_manager.cc:1875] T ace17d84b75c4ebe8f2c9d43b75a0087 P 0d6e5d3dac474d61a6960acc0b360dbc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 02:10:25.717216 15989 ts_tablet_manager.cc:1888] T ace17d84b75c4ebe8f2c9d43b75a0087 P 0d6e5d3dac474d61a6960acc0b360dbc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240507 02:10:25.717284 15989 log.cc:1196] T ace17d84b75c4ebe8f2c9d43b75a0087 P 0d6e5d3dac474d61a6960acc0b360dbc: Deleting WAL directory at /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-0/wal/wals/ace17d84b75c4ebe8f2c9d43b75a0087
I20240507 02:10:25.717499 15989 ts_tablet_manager.cc:1909] T ace17d84b75c4ebe8f2c9d43b75a0087 P 0d6e5d3dac474d61a6960acc0b360dbc: Deleting consensus metadata
I20240507 02:10:25.717957 15778 catalog_manager.cc:4830] TS 0d6e5d3dac474d61a6960acc0b360dbc (127.31.131.65:43457): tablet ace17d84b75c4ebe8f2c9d43b75a0087 (table test-table [id=176e92bf5d764c4d8801413348845fef]) successfully deleted
W20240507 02:10:25.770211 15984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:10:25.770380 15984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:10:25.770399 15984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:10:25.771703 15984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:10:25.771745 15984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:10:25.771768 15984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.66
I20240507 02:10:25.773231 15984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:43653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab
--principal=kudu/127.31.131.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.66:38643
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.131.66
--webserver_port=32811
--tserver_master_addrs=127.31.131.126:35343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.66
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:10:25.773387 15984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:10:25.773528 15984 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:10:25.777269 15997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:10:25.777390 15984 server_base.cc:1034] running on GCE node
W20240507 02:10:25.777324 15994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:10:25.777423 15995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:10:25.777685 15984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:10:25.777858 15984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:10:25.778985 15984 hybrid_clock.cc:648] HybridClock initialized: now 1715047825778977 us; error 28 us; skew 500 ppm
May 07 02:10:25 dist-test-slave-v0bg krb5kdc[15214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047825, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:10:25.781030 15984 init.cc:377] Logged in from keytab as kudu/127.31.131.66@KRBTEST.COM (short username kudu)
I20240507 02:10:25.781391 15984 webserver.cc:460] Webserver started at http://127.31.131.66:32811/ using document root <none> and password file <none>
I20240507 02:10:25.781553 15984 fs_manager.cc:359] Metadata directory not provided
I20240507 02:10:25.781617 15984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:10:25.782765 15984 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20240507 02:10:25.783207 16004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:10:25.783352 15984 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240507 02:10:25.783422 15984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-1/wal
uuid: "c5a8b5ba9bf94f4aa922d3d5307406b6"
format_stamp: "Formatted at 2024-05-07 02:10:05 on dist-test-slave-v0bg"
server_key: "da659846df383f31f89105bcdf6de9d0"
server_key_iv: "22baf740d3e4b3c8bcee96872b10d9c9"
server_key_version: "encryptionkey@0"
I20240507 02:10:25.783684 15984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:10:25.796308 15984 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:10:25.797740 15984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:10:25.797844 15984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:10:25.798019 15984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:10:25.798300 15984 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:10:25.798331 15984 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:10:25.798358 15984 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:10:25.798388 15984 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:10:25.803350 15984 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.66:38643
I20240507 02:10:25.803400 16117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.66:38643 every 8 connection(s)
I20240507 02:10:25.803669 15984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
May 07 02:10:25 dist-test-slave-v0bg krb5kdc[15214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047825, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.66@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:10:25.810889 15983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:25.804264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.66:40609 (local address 127.31.131.126:35343)
0507 02:10:25.804313 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0507 02:10:25.804315 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:10:25.804468 (+ 153us) server_negotiation.cc:407] Connection header received
0507 02:10:25.804901 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:10:25.804903 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:10:25.804911 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:10:25.804950 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:10:25.805916 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:10:25.806395 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:10:25.806904 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:10:25.807025 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:10:25.808740 (+ 1715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:10:25.808744 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:10:25.808744 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:10:25.808750 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:10:25.809935 (+ 1185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:10:25.810264 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:10:25.810265 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:10:25.810265 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:10:25.810283 (+ 18us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:10:25.810412 (+ 129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:10:25.810413 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:10:25.810413 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:10:25.810490 (+ 77us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:10:25.810531 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0507 02:10:25.810756 (+ 225us) server_negotiation.cc:299] Negotiation successful
0507 02:10:25.810823 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20240507 02:10:25.810973 16120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:25.804247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:35343 (local address 127.31.131.66:40609)
0507 02:10:25.804372 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:25.804395 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0507 02:10:25.804814 (+ 419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:10:25.805005 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:10:25.805008 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:10:25.805352 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:10:25.805848 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:10:25.805852 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:10:25.806453 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:10:25.806454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:10:25.806842 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:10:25.806843 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:10:25.806937 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:10:25.807373 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:10:25.807387 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:10:25.808655 (+ 1268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:10:25.810007 (+ 1352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:10:25.810012 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:10:25.810015 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:10:25.810206 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:10:25.810327 (+ 121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:10:25.810328 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:10:25.810328 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:10:25.810375 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:10:25.810528 (+ 153us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:10:25.810531 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:10:25.810656 (+ 125us) client_negotiation.cc:769] Sending connection context
0507 02:10:25.810739 (+ 83us) client_negotiation.cc:240] Negotiation successful
0507 02:10:25.810836 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":67,"threads_started":1}
I20240507 02:10:25.811381 16118 heartbeater.cc:347] Connected to a master server at 127.31.131.126:35343
I20240507 02:10:25.811478 16118 heartbeater.cc:464] Registering TS with master...
I20240507 02:10:25.811506 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 15984
I20240507 02:10:25.811681 16118 heartbeater.cc:511] Master 127.31.131.126:35343 requested a full tablet report, sending...
I20240507 02:10:25.811797 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.67:42387
--local_ip_for_outbound_sockets=127.31.131.67
--tserver_master_addrs=127.31.131.126:35343
--webserver_port=46439
--webserver_interface=127.31.131.67
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab
--principal=kudu/127.31.131.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:43653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240507 02:10:25.812072 15792 ts_manager.cc:194] Registered new tserver with Master: c5a8b5ba9bf94f4aa922d3d5307406b6 (127.31.131.66:38643)
I20240507 02:10:25.812435 15792 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.131.66@KRBTEST.COM'} at 127.31.131.66:40609
W20240507 02:10:25.889503 16121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:10:25.889673 16121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:10:25.889694 16121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:10:25.891016 16121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:10:25.891057 16121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:10:25.891080 16121 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.67
I20240507 02:10:25.892611 16121 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:43653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab
--principal=kudu/127.31.131.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.67:42387
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.131.67
--webserver_port=46439
--tserver_master_addrs=127.31.131.126:35343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.67
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:10:25.892792 16121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:10:25.892964 16121 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:10:25.896565 16126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:10:25.896664 16121 server_base.cc:1034] running on GCE node
W20240507 02:10:25.896659 16129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:10:25.896785 16127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:10:25.896961 16121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:10:25.897136 16121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:10:25.898259 16121 hybrid_clock.cc:648] HybridClock initialized: now 1715047825898251 us; error 25 us; skew 500 ppm
May 07 02:10:25 dist-test-slave-v0bg krb5kdc[15214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047825, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:10:25.900343 16121 init.cc:377] Logged in from keytab as kudu/127.31.131.67@KRBTEST.COM (short username kudu)
I20240507 02:10:25.900688 16121 webserver.cc:460] Webserver started at http://127.31.131.67:46439/ using document root <none> and password file <none>
I20240507 02:10:25.900837 16121 fs_manager.cc:359] Metadata directory not provided
I20240507 02:10:25.900883 16121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:10:25.901978 16121 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20240507 02:10:25.902418 16136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:10:25.902596 16121 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240507 02:10:25.902669 16121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-2/wal
uuid: "da19d23f7add4b50bf53330de753ce77"
format_stamp: "Formatted at 2024-05-07 02:10:05 on dist-test-slave-v0bg"
server_key: "0bad5911b238d5d777b5e092a8e2d14b"
server_key_iv: "ddb757c23556d4f449a3d2b79f605e33"
server_key_version: "encryptionkey@0"
I20240507 02:10:25.902904 16121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:10:25.914975 16121 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:10:25.916402 16121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:10:25.916486 16121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:10:25.916627 16121 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:10:25.916844 16121 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:10:25.916872 16121 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:10:25.916890 16121 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:10:25.916906 16121 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:10:25.921563 16121 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.67:42387
I20240507 02:10:25.921633 16249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.67:42387 every 8 connection(s)
I20240507 02:10:25.921854 16121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
I20240507 02:10:25.925815 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 16121
May 07 02:10:25 dist-test-slave-v0bg krb5kdc[15214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047825, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.67@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:10:25.929452 15983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:25.922424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.67:58989 (local address 127.31.131.126:35343)
0507 02:10:25.922468 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0507 02:10:25.922470 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:10:25.922644 (+ 174us) server_negotiation.cc:407] Connection header received
0507 02:10:25.923078 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:10:25.923079 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:10:25.923089 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:10:25.923140 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:10:25.924000 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:10:25.924471 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:10:25.924945 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:10:25.925045 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:10:25.926946 (+ 1901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:10:25.926950 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:10:25.926951 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:10:25.926958 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:10:25.928371 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:10:25.928736 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:10:25.928737 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:10:25.928738 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:10:25.928764 (+ 26us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:10:25.928933 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:10:25.928934 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:10:25.928934 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:10:25.929010 (+ 76us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:10:25.929064 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0507 02:10:25.929277 (+ 213us) server_negotiation.cc:299] Negotiation successful
0507 02:10:25.929360 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20240507 02:10:25.929535 16252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:25.922386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:35343 (local address 127.31.131.67:58989)
0507 02:10:25.922524 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:25.922569 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0507 02:10:25.922999 (+ 430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:10:25.923183 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:10:25.923186 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:10:25.923438 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:10:25.923932 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:10:25.923936 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:10:25.924529 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:10:25.924530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:10:25.924892 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:10:25.924893 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:10:25.924981 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:10:25.925407 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:10:25.925421 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:10:25.926865 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:10:25.928452 (+ 1587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:10:25.928456 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:10:25.928458 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:10:25.928664 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:10:25.928824 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:10:25.928825 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:10:25.928825 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:10:25.928876 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:10:25.929060 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:10:25.929064 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:10:25.929165 (+ 101us) client_negotiation.cc:769] Sending connection context
0507 02:10:25.929251 (+ 86us) client_negotiation.cc:240] Negotiation successful
0507 02:10:25.929366 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":81,"threads_started":1}
I20240507 02:10:25.929968 16250 heartbeater.cc:347] Connected to a master server at 127.31.131.126:35343
I20240507 02:10:25.930076 16250 heartbeater.cc:464] Registering TS with master...
I20240507 02:10:25.930289 16250 heartbeater.cc:511] Master 127.31.131.126:35343 requested a full tablet report, sending...
I20240507 02:10:25.930732 15792 ts_manager.cc:194] Registered new tserver with Master: da19d23f7add4b50bf53330de753ce77 (127.31.131.67:42387)
I20240507 02:10:25.931087 15792 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.131.67@KRBTEST.COM'} at 127.31.131.67:58989
I20240507 02:10:25.931926 16254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:25.926181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60550 (local address 127.31.131.126:35343)
0507 02:10:25.926534 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0507 02:10:25.926536 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:10:25.926543 (+ 7us) server_negotiation.cc:407] Connection header received
0507 02:10:25.926584 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:10:25.926585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:10:25.926595 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:10:25.926656 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:10:25.927404 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:10:25.927880 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:10:25.928622 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:10:25.928774 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:10:25.929576 (+ 802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:10:25.929580 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:10:25.929580 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:10:25.929586 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:10:25.931023 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:10:25.931381 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:10:25.931382 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:10:25.931382 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:10:25.931403 (+ 21us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:10:25.931542 (+ 139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:10:25.931542 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:10:25.931542 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:10:25.931608 (+ 66us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:10:25.931701 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0507 02:10:25.931806 (+ 105us) server_negotiation.cc:299] Negotiation successful
0507 02:10:25.931866 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":55,"threads_started":1}
I20240507 02:10:25.932528 32269 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240507 02:10:25.934480 16254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:25.932818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60558 (local address 127.31.131.126:35343)
0507 02:10:25.932860 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0507 02:10:25.932861 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:10:25.932948 (+ 87us) server_negotiation.cc:407] Connection header received
0507 02:10:25.932994 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:10:25.932995 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:10:25.933013 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:10:25.933069 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0507 02:10:25.933259 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:10:25.933670 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:10:25.934133 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:10:25.934227 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:10:25.934254 (+ 27us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0507 02:10:25.934333 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0507 02:10:25.934382 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0507 02:10:25.934423 (+ 41us) server_negotiation.cc:299] Negotiation successful
0507 02:10:25.934439 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20240507 02:10:25.934752 15792 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:60558:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 02:10:25.936092 15792 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 02:10:25.938025 15914 tablet_service.cc:1467] Processing CreateTablet for tablet 1aca4532ebbe437d9839a2dd92e667bc (DEFAULT_TABLE table=test-table [id=a42f41007dd04a2ea572a54a6c17df63]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:10:25.938220 15914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1aca4532ebbe437d9839a2dd92e667bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:10:25.939653 15872 tablet_bootstrap.cc:492] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc: Bootstrap starting.
I20240507 02:10:25.939972 15872 tablet_bootstrap.cc:654] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc: Neither blocks nor log segments found. Creating new log.
I20240507 02:10:25.940321 15987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:25.937622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.67:42387 (local address 127.0.0.1:39790)
0507 02:10:25.937819 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:25.937828 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0507 02:10:25.937860 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:10:25.938070 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:10:25.938071 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:10:25.938072 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:10:25.938225 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:10:25.938227 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:10:25.939326 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:10:25.939327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:10:25.940097 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:10:25.940100 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:10:25.940158 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:10:25.940189 (+ 31us) client_negotiation.cc:769] Sending connection context
0507 02:10:25.940221 (+ 32us) client_negotiation.cc:240] Negotiation successful
0507 02:10:25.940258 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":9,"thread_start_us":93,"threads_started":1}
I20240507 02:10:25.941066 16257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:25.937701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39790 (local address 127.31.131.67:42387)
0507 02:10:25.937932 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0507 02:10:25.937938 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0507 02:10:25.937945 (+ 7us) server_negotiation.cc:407] Connection header received
0507 02:10:25.937967 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:10:25.937968 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:10:25.937979 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:10:25.938038 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:10:25.938307 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:10:25.939250 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:10:25.940423 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:10:25.940901 (+ 478us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:10:25.940944 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0507 02:10:25.940970 (+ 26us) server_negotiation.cc:299] Negotiation successful
0507 02:10:25.941000 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":76,"threads_started":1}
I20240507 02:10:25.941630 15872 tablet_bootstrap.cc:492] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc: No bootstrap required, opened a new log
I20240507 02:10:25.941550 16184 tablet_service.cc:1467] Processing CreateTablet for tablet 1aca4532ebbe437d9839a2dd92e667bc (DEFAULT_TABLE table=test-table [id=a42f41007dd04a2ea572a54a6c17df63]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:10:25.941685 15872 ts_tablet_manager.cc:1367] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240507 02:10:25.941804 16184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1aca4532ebbe437d9839a2dd92e667bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:10:25.941783 15872 raft_consensus.cc:348] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a8b5ba9bf94f4aa922d3d5307406b6" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 38643 } } peers { permanent_uuid: "da19d23f7add4b50bf53330de753ce77" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 42387 } } peers { permanent_uuid: "0d6e5d3dac474d61a6960acc0b360dbc" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43457 } }
I20240507 02:10:25.941844 15872 raft_consensus.cc:374] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:10:25.941864 15872 raft_consensus.cc:729] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d6e5d3dac474d61a6960acc0b360dbc, State: Initialized, Role: FOLLOWER
I20240507 02:10:25.941907 15872 consensus_queue.cc:260] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a8b5ba9bf94f4aa922d3d5307406b6" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 38643 } } peers { permanent_uuid: "da19d23f7add4b50bf53330de753ce77" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 42387 } } peers { permanent_uuid: "0d6e5d3dac474d61a6960acc0b360dbc" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43457 } }
I20240507 02:10:25.941989 15872 ts_tablet_manager.cc:1398] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20240507 02:10:25.942848 16256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:25.937621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.66:38643 (local address 127.0.0.1:53278)
0507 02:10:25.937765 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:25.937772 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0507 02:10:25.937827 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:10:25.939895 (+ 2068us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:10:25.939896 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:10:25.939898 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:10:25.940076 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:10:25.940079 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:10:25.941278 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:10:25.941279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:10:25.941992 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:10:25.941995 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:10:25.942699 (+ 704us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:10:25.942703 (+ 4us) client_negotiation.cc:769] Sending connection context
0507 02:10:25.942747 (+ 44us) client_negotiation.cc:240] Negotiation successful
0507 02:10:25.942793 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129}
I20240507 02:10:25.942886 16258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:25.937848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53278 (local address 127.31.131.66:38643)
0507 02:10:25.939721 (+ 1873us) server_negotiation.cc:206] Beginning negotiation
0507 02:10:25.939728 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0507 02:10:25.939739 (+ 11us) server_negotiation.cc:407] Connection header received
0507 02:10:25.939777 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:10:25.939778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:10:25.939793 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:10:25.939874 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:10:25.940157 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:10:25.941203 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:10:25.942107 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:10:25.942625 (+ 518us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:10:25.942671 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0507 02:10:25.942778 (+ 107us) server_negotiation.cc:299] Negotiation successful
0507 02:10:25.942817 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1837,"thread_start_us":80,"threads_started":1}
I20240507 02:10:25.943343 16052 tablet_service.cc:1467] Processing CreateTablet for tablet 1aca4532ebbe437d9839a2dd92e667bc (DEFAULT_TABLE table=test-table [id=a42f41007dd04a2ea572a54a6c17df63]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:10:25.943565 16052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1aca4532ebbe437d9839a2dd92e667bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:10:25.944319 16260 tablet_bootstrap.cc:492] T 1aca4532ebbe437d9839a2dd92e667bc P da19d23f7add4b50bf53330de753ce77: Bootstrap starting.
I20240507 02:10:25.944870 16260 tablet_bootstrap.cc:654] T 1aca4532ebbe437d9839a2dd92e667bc P da19d23f7add4b50bf53330de753ce77: Neither blocks nor log segments found. Creating new log.
I20240507 02:10:25.945151 16260 log.cc:824] T 1aca4532ebbe437d9839a2dd92e667bc P da19d23f7add4b50bf53330de753ce77: Log is configured to *not* fsync() on all Append() calls
I20240507 02:10:25.945854 16260 tablet_bootstrap.cc:492] T 1aca4532ebbe437d9839a2dd92e667bc P da19d23f7add4b50bf53330de753ce77: No bootstrap required, opened a new log
I20240507 02:10:25.945952 16260 ts_tablet_manager.cc:1367] T 1aca4532ebbe437d9839a2dd92e667bc P da19d23f7add4b50bf53330de753ce77: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240507 02:10:25.946166 16262 tablet_bootstrap.cc:492] T 1aca4532ebbe437d9839a2dd92e667bc P c5a8b5ba9bf94f4aa922d3d5307406b6: Bootstrap starting.
I20240507 02:10:25.946663 16262 tablet_bootstrap.cc:654] T 1aca4532ebbe437d9839a2dd92e667bc P c5a8b5ba9bf94f4aa922d3d5307406b6: Neither blocks nor log segments found. Creating new log.
I20240507 02:10:25.946882 16262 log.cc:824] T 1aca4532ebbe437d9839a2dd92e667bc P c5a8b5ba9bf94f4aa922d3d5307406b6: Log is configured to *not* fsync() on all Append() calls
I20240507 02:10:25.947471 16262 tablet_bootstrap.cc:492] T 1aca4532ebbe437d9839a2dd92e667bc P c5a8b5ba9bf94f4aa922d3d5307406b6: No bootstrap required, opened a new log
I20240507 02:10:25.947516 16262 ts_tablet_manager.cc:1367] T 1aca4532ebbe437d9839a2dd92e667bc P c5a8b5ba9bf94f4aa922d3d5307406b6: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20240507 02:10:25.947746 16260 raft_consensus.cc:348] T 1aca4532ebbe437d9839a2dd92e667bc P da19d23f7add4b50bf53330de753ce77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a8b5ba9bf94f4aa922d3d5307406b6" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 38643 } } peers { permanent_uuid: "da19d23f7add4b50bf53330de753ce77" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 42387 } } peers { permanent_uuid: "0d6e5d3dac474d61a6960acc0b360dbc" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43457 } }
I20240507 02:10:25.947849 16260 raft_consensus.cc:374] T 1aca4532ebbe437d9839a2dd92e667bc P da19d23f7add4b50bf53330de753ce77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:10:25.947870 16260 raft_consensus.cc:729] T 1aca4532ebbe437d9839a2dd92e667bc P da19d23f7add4b50bf53330de753ce77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da19d23f7add4b50bf53330de753ce77, State: Initialized, Role: FOLLOWER
I20240507 02:10:25.947950 16260 consensus_queue.cc:260] T 1aca4532ebbe437d9839a2dd92e667bc P da19d23f7add4b50bf53330de753ce77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a8b5ba9bf94f4aa922d3d5307406b6" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 38643 } } peers { permanent_uuid: "da19d23f7add4b50bf53330de753ce77" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 42387 } } peers { permanent_uuid: "0d6e5d3dac474d61a6960acc0b360dbc" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43457 } }
I20240507 02:10:25.948163 16260 ts_tablet_manager.cc:1398] T 1aca4532ebbe437d9839a2dd92e667bc P da19d23f7add4b50bf53330de753ce77: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240507 02:10:25.948202 16250 heartbeater.cc:503] Master 127.31.131.126:35343 was elected leader, sending a full tablet report...
I20240507 02:10:25.949355 16262 raft_consensus.cc:348] T 1aca4532ebbe437d9839a2dd92e667bc P c5a8b5ba9bf94f4aa922d3d5307406b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a8b5ba9bf94f4aa922d3d5307406b6" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 38643 } } peers { permanent_uuid: "da19d23f7add4b50bf53330de753ce77" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 42387 } } peers { permanent_uuid: "0d6e5d3dac474d61a6960acc0b360dbc" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43457 } }
I20240507 02:10:25.949471 16262 raft_consensus.cc:374] T 1aca4532ebbe437d9839a2dd92e667bc P c5a8b5ba9bf94f4aa922d3d5307406b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:10:25.949503 16262 raft_consensus.cc:729] T 1aca4532ebbe437d9839a2dd92e667bc P c5a8b5ba9bf94f4aa922d3d5307406b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5a8b5ba9bf94f4aa922d3d5307406b6, State: Initialized, Role: FOLLOWER
I20240507 02:10:25.949592 16262 consensus_queue.cc:260] T 1aca4532ebbe437d9839a2dd92e667bc P c5a8b5ba9bf94f4aa922d3d5307406b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a8b5ba9bf94f4aa922d3d5307406b6" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 38643 } } peers { permanent_uuid: "da19d23f7add4b50bf53330de753ce77" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 42387 } } peers { permanent_uuid: "0d6e5d3dac474d61a6960acc0b360dbc" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43457 } }
I20240507 02:10:25.949793 16262 ts_tablet_manager.cc:1398] T 1aca4532ebbe437d9839a2dd92e667bc P c5a8b5ba9bf94f4aa922d3d5307406b6: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240507 02:10:25.949851 16118 heartbeater.cc:503] Master 127.31.131.126:35343 was elected leader, sending a full tablet report...
I20240507 02:10:25.985460 15986 raft_consensus.cc:482] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 02:10:25.985610 15986 raft_consensus.cc:504] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a8b5ba9bf94f4aa922d3d5307406b6" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 38643 } } peers { permanent_uuid: "da19d23f7add4b50bf53330de753ce77" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 42387 } } peers { permanent_uuid: "0d6e5d3dac474d61a6960acc0b360dbc" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43457 } }
I20240507 02:10:25.985908 15986 leader_election.cc:288] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5a8b5ba9bf94f4aa922d3d5307406b6 (127.31.131.66:38643), da19d23f7add4b50bf53330de753ce77 (127.31.131.67:42387)
I20240507 02:10:25.988252 15982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:25.986060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.66:38643 (local address 127.31.131.65:54559)
0507 02:10:25.986117 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:25.986129 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0507 02:10:25.986191 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:10:25.986345 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:10:25.986346 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:10:25.986349 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:10:25.986539 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:10:25.986541 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:10:25.987377 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:10:25.987378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:10:25.988075 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:10:25.988077 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:10:25.988134 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:10:25.988156 (+ 22us) client_negotiation.cc:769] Sending connection context
0507 02:10:25.988176 (+ 20us) client_negotiation.cc:240] Negotiation successful
0507 02:10:25.988201 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20240507 02:10:25.988716 16258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:25.986138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:54559 (local address 127.31.131.66:38643)
0507 02:10:25.986186 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0507 02:10:25.986188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:10:25.986196 (+ 8us) server_negotiation.cc:407] Connection header received
0507 02:10:25.986274 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:10:25.986275 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:10:25.986284 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:10:25.986335 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:10:25.986631 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:10:25.987313 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:10:25.988143 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:10:25.988580 (+ 437us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:10:25.988589 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0507 02:10:25.988622 (+ 33us) server_negotiation.cc:299] Negotiation successful
0507 02:10:25.988655 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20240507 02:10:25.988754 16266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:25.986060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.67:42387 (local address 127.31.131.65:37437)
0507 02:10:25.986329 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:25.986337 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0507 02:10:25.986389 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:10:25.986499 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:10:25.986500 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:10:25.986501 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:10:25.987006 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:10:25.987008 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:10:25.987822 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:10:25.987823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:10:25.988549 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:10:25.988551 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:10:25.988623 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:10:25.988627 (+ 4us) client_negotiation.cc:769] Sending connection context
0507 02:10:25.988655 (+ 28us) client_negotiation.cc:240] Negotiation successful
0507 02:10:25.988688 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":73,"thread_start_us":115,"threads_started":1}
I20240507 02:10:25.988968 16072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1aca4532ebbe437d9839a2dd92e667bc" candidate_uuid: "0d6e5d3dac474d61a6960acc0b360dbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5a8b5ba9bf94f4aa922d3d5307406b6" is_pre_election: true
I20240507 02:10:25.989086 16072 raft_consensus.cc:2454] T 1aca4532ebbe437d9839a2dd92e667bc P c5a8b5ba9bf94f4aa922d3d5307406b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d6e5d3dac474d61a6960acc0b360dbc in term 0.
I20240507 02:10:25.989120 16257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:25.986169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:37437 (local address 127.31.131.67:42387)
0507 02:10:25.986212 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0507 02:10:25.986214 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:10:25.986398 (+ 184us) server_negotiation.cc:407] Connection header received
0507 02:10:25.986428 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:10:25.986429 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:10:25.986438 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:10:25.986494 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:10:25.987085 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:10:25.987733 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:10:25.988634 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:10:25.989015 (+ 381us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:10:25.989022 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0507 02:10:25.989040 (+ 18us) server_negotiation.cc:299] Negotiation successful
0507 02:10:25.989067 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20240507 02:10:25.989287 15869 leader_election.cc:302] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d6e5d3dac474d61a6960acc0b360dbc, c5a8b5ba9bf94f4aa922d3d5307406b6; no voters:
I20240507 02:10:25.989401 16204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1aca4532ebbe437d9839a2dd92e667bc" candidate_uuid: "0d6e5d3dac474d61a6960acc0b360dbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da19d23f7add4b50bf53330de753ce77" is_pre_election: true
I20240507 02:10:25.989424 15986 raft_consensus.cc:2781] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 02:10:25.989487 15986 raft_consensus.cc:482] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 02:10:25.989508 16204 raft_consensus.cc:2454] T 1aca4532ebbe437d9839a2dd92e667bc P da19d23f7add4b50bf53330de753ce77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d6e5d3dac474d61a6960acc0b360dbc in term 0.
I20240507 02:10:25.989518 15986 raft_consensus.cc:3037] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:10:25.990164 15986 raft_consensus.cc:504] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a8b5ba9bf94f4aa922d3d5307406b6" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 38643 } } peers { permanent_uuid: "da19d23f7add4b50bf53330de753ce77" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 42387 } } peers { permanent_uuid: "0d6e5d3dac474d61a6960acc0b360dbc" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43457 } }
I20240507 02:10:25.990289 15986 leader_election.cc:288] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc [CANDIDATE]: Term 1 election: Requested vote from peers c5a8b5ba9bf94f4aa922d3d5307406b6 (127.31.131.66:38643), da19d23f7add4b50bf53330de753ce77 (127.31.131.67:42387)
I20240507 02:10:25.990411 16204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1aca4532ebbe437d9839a2dd92e667bc" candidate_uuid: "0d6e5d3dac474d61a6960acc0b360dbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da19d23f7add4b50bf53330de753ce77"
I20240507 02:10:25.990411 16072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1aca4532ebbe437d9839a2dd92e667bc" candidate_uuid: "0d6e5d3dac474d61a6960acc0b360dbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5a8b5ba9bf94f4aa922d3d5307406b6"
I20240507 02:10:25.990485 16072 raft_consensus.cc:3037] T 1aca4532ebbe437d9839a2dd92e667bc P c5a8b5ba9bf94f4aa922d3d5307406b6 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:10:25.990485 16204 raft_consensus.cc:3037] T 1aca4532ebbe437d9839a2dd92e667bc P da19d23f7add4b50bf53330de753ce77 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:10:25.991331 16072 raft_consensus.cc:2454] T 1aca4532ebbe437d9839a2dd92e667bc P c5a8b5ba9bf94f4aa922d3d5307406b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d6e5d3dac474d61a6960acc0b360dbc in term 1.
I20240507 02:10:25.991331 16204 raft_consensus.cc:2454] T 1aca4532ebbe437d9839a2dd92e667bc P da19d23f7add4b50bf53330de753ce77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d6e5d3dac474d61a6960acc0b360dbc in term 1.
I20240507 02:10:25.991506 15868 leader_election.cc:302] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d6e5d3dac474d61a6960acc0b360dbc, da19d23f7add4b50bf53330de753ce77; no voters:
I20240507 02:10:25.991617 15986 raft_consensus.cc:2781] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:10:25.991742 15986 raft_consensus.cc:686] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc [term 1 LEADER]: Becoming Leader. State: Replica: 0d6e5d3dac474d61a6960acc0b360dbc, State: Running, Role: LEADER
I20240507 02:10:25.991797 15986 consensus_queue.cc:237] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a8b5ba9bf94f4aa922d3d5307406b6" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 38643 } } peers { permanent_uuid: "da19d23f7add4b50bf53330de753ce77" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 42387 } } peers { permanent_uuid: "0d6e5d3dac474d61a6960acc0b360dbc" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43457 } }
I20240507 02:10:25.992476 15792 catalog_manager.cc:5484] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d6e5d3dac474d61a6960acc0b360dbc (127.31.131.65). New cstate: current_term: 1 leader_uuid: "0d6e5d3dac474d61a6960acc0b360dbc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a8b5ba9bf94f4aa922d3d5307406b6" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 38643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da19d23f7add4b50bf53330de753ce77" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 42387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d6e5d3dac474d61a6960acc0b360dbc" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43457 } health_report { overall_health: HEALTHY } } }
I20240507 02:10:25.998035 15988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:25.996281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60914 (local address 127.31.131.65:43457)
0507 02:10:25.996332 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0507 02:10:25.996334 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:10:25.996341 (+ 7us) server_negotiation.cc:407] Connection header received
0507 02:10:25.996373 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:10:25.996373 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:10:25.996390 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:10:25.996447 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0507 02:10:25.996638 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:10:25.997073 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:10:25.997585 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:10:25.997727 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:10:25.997759 (+ 32us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0507 02:10:25.997832 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0507 02:10:25.997893 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0507 02:10:25.997945 (+ 52us) server_negotiation.cc:299] Negotiation successful
0507 02:10:25.997967 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20240507 02:10:25.999202 16204 raft_consensus.cc:1261] T 1aca4532ebbe437d9839a2dd92e667bc P da19d23f7add4b50bf53330de753ce77 [term 1 FOLLOWER]: Refusing update from remote peer 0d6e5d3dac474d61a6960acc0b360dbc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 02:10:25.999202 16072 raft_consensus.cc:1261] T 1aca4532ebbe437d9839a2dd92e667bc P c5a8b5ba9bf94f4aa922d3d5307406b6 [term 1 FOLLOWER]: Refusing update from remote peer 0d6e5d3dac474d61a6960acc0b360dbc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 02:10:25.999475 16267 consensus_queue.cc:1035] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5a8b5ba9bf94f4aa922d3d5307406b6" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 38643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 02:10:25.999598 15986 consensus_queue.cc:1035] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc [LEADER]: Connected to new peer: Peer: permanent_uuid: "da19d23f7add4b50bf53330de753ce77" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 42387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 02:10:26.002234 16271 mvcc.cc:205] Tried to move back new op lower bound from 7024835895290945536 to 7024835895263072256. Current Snapshot: MvccSnapshot[applied={T|T < 7024835895290945536 or (T in {7024835895290945536})}]
I20240507 02:10:26.003209 15792 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:60558:
table { table_name: "test-table" } modify_external_catalogs: true
I20240507 02:10:26.003289 15792 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:60558:
table { table_name: "test-table" } modify_external_catalogs: true
I20240507 02:10:26.003902 15792 catalog_manager.cc:5771] T 00000000000000000000000000000000 P dec8ad1f575d48a59f9845b310f4af5b: Sending DeleteTablet for 3 replicas of tablet 1aca4532ebbe437d9839a2dd92e667bc
I20240507 02:10:26.004168 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 15830
I20240507 02:10:26.004171 15914 tablet_service.cc:1514] Processing DeleteTablet for tablet 1aca4532ebbe437d9839a2dd92e667bc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:10:26 UTC) from {username='kudu', principal='kudu/127.31.131.126@KRBTEST.COM'} at 127.0.0.1:60910
I20240507 02:10:26.004186 16052 tablet_service.cc:1514] Processing DeleteTablet for tablet 1aca4532ebbe437d9839a2dd92e667bc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:10:26 UTC) from {username='kudu', principal='kudu/127.31.131.126@KRBTEST.COM'} at 127.0.0.1:53278
I20240507 02:10:26.004709 16280 tablet_replica.cc:330] T 1aca4532ebbe437d9839a2dd92e667bc P c5a8b5ba9bf94f4aa922d3d5307406b6: stopping tablet replica
I20240507 02:10:26.004781 16280 raft_consensus.cc:2229] T 1aca4532ebbe437d9839a2dd92e667bc P c5a8b5ba9bf94f4aa922d3d5307406b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 02:10:26.004842 16280 raft_consensus.cc:2258] T 1aca4532ebbe437d9839a2dd92e667bc P c5a8b5ba9bf94f4aa922d3d5307406b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 02:10:26.004935 16184 tablet_service.cc:1514] Processing DeleteTablet for tablet 1aca4532ebbe437d9839a2dd92e667bc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:10:26 UTC) from {username='kudu', principal='kudu/127.31.131.126@KRBTEST.COM'} at 127.0.0.1:39790
I20240507 02:10:26.005136 16281 tablet_replica.cc:330] T 1aca4532ebbe437d9839a2dd92e667bc P da19d23f7add4b50bf53330de753ce77: stopping tablet replica
I20240507 02:10:26.005199 16281 raft_consensus.cc:2229] T 1aca4532ebbe437d9839a2dd92e667bc P da19d23f7add4b50bf53330de753ce77 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 02:10:26.005301 16281 raft_consensus.cc:2258] T 1aca4532ebbe437d9839a2dd92e667bc P da19d23f7add4b50bf53330de753ce77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 02:10:26.005391 16280 ts_tablet_manager.cc:1875] T 1aca4532ebbe437d9839a2dd92e667bc P c5a8b5ba9bf94f4aa922d3d5307406b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 02:10:26.005650 16281 ts_tablet_manager.cc:1875] T 1aca4532ebbe437d9839a2dd92e667bc P da19d23f7add4b50bf53330de753ce77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 02:10:26.006856 16280 ts_tablet_manager.cc:1888] T 1aca4532ebbe437d9839a2dd92e667bc P c5a8b5ba9bf94f4aa922d3d5307406b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240507 02:10:26.006927 16280 log.cc:1196] T 1aca4532ebbe437d9839a2dd92e667bc P c5a8b5ba9bf94f4aa922d3d5307406b6: Deleting WAL directory at /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-1/wal/wals/1aca4532ebbe437d9839a2dd92e667bc
I20240507 02:10:26.007023 16281 ts_tablet_manager.cc:1888] T 1aca4532ebbe437d9839a2dd92e667bc P da19d23f7add4b50bf53330de753ce77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240507 02:10:26.007117 16281 log.cc:1196] T 1aca4532ebbe437d9839a2dd92e667bc P da19d23f7add4b50bf53330de753ce77: Deleting WAL directory at /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-2/wal/wals/1aca4532ebbe437d9839a2dd92e667bc
I20240507 02:10:26.007246 16280 ts_tablet_manager.cc:1909] T 1aca4532ebbe437d9839a2dd92e667bc P c5a8b5ba9bf94f4aa922d3d5307406b6: Deleting consensus metadata
I20240507 02:10:26.007396 16281 ts_tablet_manager.cc:1909] T 1aca4532ebbe437d9839a2dd92e667bc P da19d23f7add4b50bf53330de753ce77: Deleting consensus metadata
I20240507 02:10:26.007588 15779 catalog_manager.cc:4830] TS c5a8b5ba9bf94f4aa922d3d5307406b6 (127.31.131.66:38643): tablet 1aca4532ebbe437d9839a2dd92e667bc (table test-table [id=a42f41007dd04a2ea572a54a6c17df63]) successfully deleted
I20240507 02:10:26.007699 15778 catalog_manager.cc:4830] TS da19d23f7add4b50bf53330de753ce77 (127.31.131.67:42387): tablet 1aca4532ebbe437d9839a2dd92e667bc (table test-table [id=a42f41007dd04a2ea572a54a6c17df63]) successfully deleted
W20240507 02:10:26.009879 15778 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.131.65:43457 (error 108)
I20240507 02:10:26.010110 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 15984
I20240507 02:10:26.010625 16256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:26.010180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.65:43457 (local address 127.0.0.1:60924)
0507 02:10:26.010566 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:26.010596 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.65:43457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349}
W20240507 02:10:26.010843 15778 catalog_manager.cc:4573] TS 0d6e5d3dac474d61a6960acc0b360dbc (127.31.131.65:43457): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1aca4532ebbe437d9839a2dd92e667bc: Network error: Client connection negotiation failed: client connection to 127.31.131.65:43457: connect: Connection refused (error 111)
I20240507 02:10:26.014262 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 16121
I20240507 02:10:26.018896 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 15759
I20240507 02:10:46.023370 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:35343
--webserver_interface=127.31.131.126
--webserver_port=41271
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab
--principal=kudu/127.31.131.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:43653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.131.126:35343
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:10:46.098189 16283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:10:46.098343 16283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:10:46.098362 16283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:10:46.099630 16283 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20240507 02:10:46.099670 16283 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20240507 02:10:46.099681 16283 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20240507 02:10:46.099691 16283 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240507 02:10:46.099716 16283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:10:46.099737 16283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:10:46.099759 16283 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240507 02:10:46.099781 16283 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240507 02:10:46.101320 16283 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:43653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.31.131.126:35343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab
--principal=kudu/127.31.131.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:35343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.131.126
--webserver_port=41271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16283
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:10:46.101485 16283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:10:46.101648 16283 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:10:46.105209 16288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:10:46.105252 16283 server_base.cc:1034] running on GCE node
W20240507 02:10:46.105157 16291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:10:46.105340 16289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:10:46.105602 16283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:10:46.105782 16283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:10:46.106899 16283 hybrid_clock.cc:648] HybridClock initialized: now 1715047846106889 us; error 29 us; skew 500 ppm
May 07 02:10:46 dist-test-slave-v0bg krb5kdc[15214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047846, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:10:46.113891 16283 init.cc:377] Logged in from keytab as kudu/127.31.131.126@KRBTEST.COM (short username kudu)
I20240507 02:10:46.114285 16283 webserver.cc:460] Webserver started at http://127.31.131.126:41271/ using document root <none> and password file <none>
I20240507 02:10:46.114449 16283 fs_manager.cc:359] Metadata directory not provided
I20240507 02:10:46.114485 16283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:10:46.115598 16283 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20240507 02:10:46.116104 16298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:10:46.116247 16283 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240507 02:10:46.116309 16283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/master-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/master-0/wal
uuid: "dec8ad1f575d48a59f9845b310f4af5b"
format_stamp: "Formatted at 2024-05-07 02:10:04 on dist-test-slave-v0bg"
server_key: "0b9180c36ad370f362912ab9d5c00dbc"
server_key_iv: "e25f55c88aa80c06a136d70722054126"
server_key_version: "encryptionkey@0"
I20240507 02:10:46.116568 16283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:10:46.134744 16283 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:10:46.136157 16283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:10:46.136248 16283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:10:46.139513 16283 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.126:35343
I20240507 02:10:46.139544 16350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.126:35343 every 8 connection(s)
I20240507 02:10:46.139837 16283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
I20240507 02:10:46.140956 16351 sys_catalog.cc:263] Verifying existing consensus state
I20240507 02:10:46.141498 16351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dec8ad1f575d48a59f9845b310f4af5b: Bootstrap starting.
I20240507 02:10:46.147606 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 16283
I20240507 02:10:46.147827 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.65:43457
--local_ip_for_outbound_sockets=127.31.131.65
--tserver_master_addrs=127.31.131.126:35343
--webserver_port=40345
--webserver_interface=127.31.131.65
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab
--principal=kudu/127.31.131.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:43653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240507 02:10:46.150203 16351 log.cc:824] T 00000000000000000000000000000000 P dec8ad1f575d48a59f9845b310f4af5b: Log is configured to *not* fsync() on all Append() calls
I20240507 02:10:46.152364 16351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dec8ad1f575d48a59f9845b310f4af5b: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240507 02:10:46.152606 16351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dec8ad1f575d48a59f9845b310f4af5b: Bootstrap complete.
I20240507 02:10:46.154446 16351 raft_consensus.cc:348] T 00000000000000000000000000000000 P dec8ad1f575d48a59f9845b310f4af5b [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec8ad1f575d48a59f9845b310f4af5b" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 35343 } }
I20240507 02:10:46.154676 16351 raft_consensus.cc:729] T 00000000000000000000000000000000 P dec8ad1f575d48a59f9845b310f4af5b [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec8ad1f575d48a59f9845b310f4af5b, State: Initialized, Role: FOLLOWER
I20240507 02:10:46.154800 16351 consensus_queue.cc:260] T 00000000000000000000000000000000 P dec8ad1f575d48a59f9845b310f4af5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec8ad1f575d48a59f9845b310f4af5b" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 35343 } }
I20240507 02:10:46.154870 16351 raft_consensus.cc:388] T 00000000000000000000000000000000 P dec8ad1f575d48a59f9845b310f4af5b [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 02:10:46.154906 16351 raft_consensus.cc:482] T 00000000000000000000000000000000 P dec8ad1f575d48a59f9845b310f4af5b [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 02:10:46.154947 16351 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dec8ad1f575d48a59f9845b310f4af5b [term 2 FOLLOWER]: Advancing to term 3
I20240507 02:10:46.155757 16351 raft_consensus.cc:504] T 00000000000000000000000000000000 P dec8ad1f575d48a59f9845b310f4af5b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec8ad1f575d48a59f9845b310f4af5b" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 35343 } }
I20240507 02:10:46.155848 16351 leader_election.cc:302] T 00000000000000000000000000000000 P dec8ad1f575d48a59f9845b310f4af5b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dec8ad1f575d48a59f9845b310f4af5b; no voters:
I20240507 02:10:46.155985 16351 leader_election.cc:288] T 00000000000000000000000000000000 P dec8ad1f575d48a59f9845b310f4af5b [CANDIDATE]: Term 3 election: Requested vote from peers
I20240507 02:10:46.156042 16355 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dec8ad1f575d48a59f9845b310f4af5b [term 3 FOLLOWER]: Leader election won for term 3
I20240507 02:10:46.156183 16351 sys_catalog.cc:564] T 00000000000000000000000000000000 P dec8ad1f575d48a59f9845b310f4af5b [sys.catalog]: configured and running, proceeding with master startup.
I20240507 02:10:46.156235 16355 raft_consensus.cc:686] T 00000000000000000000000000000000 P dec8ad1f575d48a59f9845b310f4af5b [term 3 LEADER]: Becoming Leader. State: Replica: dec8ad1f575d48a59f9845b310f4af5b, State: Running, Role: LEADER
I20240507 02:10:46.156313 16355 consensus_queue.cc:237] T 00000000000000000000000000000000 P dec8ad1f575d48a59f9845b310f4af5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec8ad1f575d48a59f9845b310f4af5b" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 35343 } }
I20240507 02:10:46.156572 16356 sys_catalog.cc:455] T 00000000000000000000000000000000 P dec8ad1f575d48a59f9845b310f4af5b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "dec8ad1f575d48a59f9845b310f4af5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec8ad1f575d48a59f9845b310f4af5b" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 35343 } } }
I20240507 02:10:46.156605 16355 sys_catalog.cc:455] T 00000000000000000000000000000000 P dec8ad1f575d48a59f9845b310f4af5b [sys.catalog]: SysCatalogTable state changed. Reason: New leader dec8ad1f575d48a59f9845b310f4af5b. Latest consensus state: current_term: 3 leader_uuid: "dec8ad1f575d48a59f9845b310f4af5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec8ad1f575d48a59f9845b310f4af5b" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 35343 } } }
I20240507 02:10:46.156733 16355 sys_catalog.cc:458] T 00000000000000000000000000000000 P dec8ad1f575d48a59f9845b310f4af5b [sys.catalog]: This master's current role is: LEADER
I20240507 02:10:46.156908 16356 sys_catalog.cc:458] T 00000000000000000000000000000000 P dec8ad1f575d48a59f9845b310f4af5b [sys.catalog]: This master's current role is: LEADER
I20240507 02:10:46.156903 16364 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 02:10:46.157686 16364 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240507 02:10:46.157830 16364 catalog_manager.cc:1259] Loaded cluster ID: 69cbc585993f46de82c1ce606effdf61
I20240507 02:10:46.158077 16364 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240507 02:10:46.159322 16364 catalog_manager.cc:1504] Loading token signing keys...
I20240507 02:10:46.159482 16364 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dec8ad1f575d48a59f9845b310f4af5b: Loaded TSK: 0
I20240507 02:10:46.159528 16364 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dec8ad1f575d48a59f9845b310f4af5b: Loaded TSK: 1
F20240507 02:10:46.159708 16364 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:asn1_check_tlen:wrong tag:../crypto/asn1/tasn_dec.c:1149 error:0D06C03A:asn1 encoding routines:asn1_d2i_ex_primitive:nested asn1 error:../crypto/asn1/tasn_dec.c:713 error:0D08303A:asn1 encoding routines:asn1_template_noexp_d2i:nested asn1 error:../crypto/asn1/tasn_dec.c:646:Field=version, Type=RSAPrivateKey error:04093004:rsa routines:old_rsa_priv_decode:RSA lib:../crypto/rsa/rsa_ameth.c:130 error:0D0680A8:asn1 encoding routines:asn1_check_tlen:wrong tag:../crypto/asn1/tasn_dec.c:1149 error:0D06C03A:asn1 encoding routines:asn1_d2i_ex_primitive:nested asn1 error:../crypto/asn1/tasn_dec.c:713 error:0D08303A:asn1 encoding routines:asn1_template_noexp_d2i:nested asn1 error:../crypto/asn1/tasn_dec.c:646:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fd2e30a0dcd google::LogMessage::Fail() at ??:0
@ 0x7fd2e30a4b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fd2e30a07cc google::LogMessage::Flush() at ??:0
@ 0x7fd2e30a1f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fd2e18f22f6 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fd2e18efbf3 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fd2e58b961e kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fd2e58b983a kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fd2e58b9983 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fd2e5879627 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fd2e5884463 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fd2e419aa08 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fd2e418f3a4 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fd2e44b26db start_thread at ??:0
@ 0x7fd2e25f571f clone at ??:0
W20240507 02:10:46.237416 16353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:10:46.237568 16353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:10:46.237587 16353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:10:46.238854 16353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:10:46.238893 16353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:10:46.238914 16353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.65
I20240507 02:10:46.240396 16353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:43653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab
--principal=kudu/127.31.131.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.65:43457
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.131.65
--webserver_port=40345
--tserver_master_addrs=127.31.131.126:35343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.65
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:10:46.240552 16353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:10:46.240700 16353 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:10:46.244159 16393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:10:46.244217 16392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:10:46.244201 16395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:10:46.244486 16353 server_base.cc:1034] running on GCE node
I20240507 02:10:46.244628 16353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:10:46.244807 16353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:10:46.245939 16353 hybrid_clock.cc:648] HybridClock initialized: now 1715047846245927 us; error 29 us; skew 500 ppm
May 07 02:10:46 dist-test-slave-v0bg krb5kdc[15214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047846, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:10:46.248020 16353 init.cc:377] Logged in from keytab as kudu/127.31.131.65@KRBTEST.COM (short username kudu)
I20240507 02:10:46.248363 16353 webserver.cc:460] Webserver started at http://127.31.131.65:40345/ using document root <none> and password file <none>
I20240507 02:10:46.248510 16353 fs_manager.cc:359] Metadata directory not provided
I20240507 02:10:46.248569 16353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:10:46.249687 16353 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20240507 02:10:46.250221 16402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:10:46.250368 16353 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240507 02:10:46.250435 16353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-0/wal
uuid: "0d6e5d3dac474d61a6960acc0b360dbc"
format_stamp: "Formatted at 2024-05-07 02:10:04 on dist-test-slave-v0bg"
server_key: "5eeb8e1ff74dba96a2a309487f9cf3ea"
server_key_iv: "33df10cbc886a1a47eeff61fa4082c42"
server_key_version: "encryptionkey@0"
I20240507 02:10:46.250730 16353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:10:46.282904 16353 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:10:46.284868 16353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:10:46.284973 16353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:10:46.285153 16353 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:10:46.285516 16409 ts_tablet_manager.cc:516] Loading tablet metadata (0/1 complete)
I20240507 02:10:46.286608 16353 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240507 02:10:46.286651 16353 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20240507 02:10:46.286682 16353 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240507 02:10:46.287485 16353 ts_tablet_manager.cc:590] Registered 1 tablets
I20240507 02:10:46.287516 16353 ts_tablet_manager.cc:569] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20240507 02:10:46.287600 16409 tablet_bootstrap.cc:492] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc: Bootstrap starting.
I20240507 02:10:46.294073 16353 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.65:43457
I20240507 02:10:46.294184 16516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.65:43457 every 8 connection(s)
I20240507 02:10:46.294448 16353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
I20240507 02:10:46.299402 16409 log.cc:824] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc: Log is configured to *not* fsync() on all Append() calls
I20240507 02:10:46.301709 16409 tablet_bootstrap.cc:492] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240507 02:10:46.301964 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 16353
I20240507 02:10:46.301980 16409 tablet_bootstrap.cc:492] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc: Bootstrap complete.
I20240507 02:10:46.302181 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.66:38643
--local_ip_for_outbound_sockets=127.31.131.66
--tserver_master_addrs=127.31.131.126:35343
--webserver_port=32811
--webserver_interface=127.31.131.66
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab
--principal=kudu/127.31.131.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:43653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240507 02:10:46.303063 16409 ts_tablet_manager.cc:1367] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.009s
I20240507 02:10:46.304244 16409 raft_consensus.cc:348] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a8b5ba9bf94f4aa922d3d5307406b6" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 38643 } } peers { permanent_uuid: "da19d23f7add4b50bf53330de753ce77" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 42387 } } peers { permanent_uuid: "0d6e5d3dac474d61a6960acc0b360dbc" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43457 } }
I20240507 02:10:46.304584 16409 raft_consensus.cc:729] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d6e5d3dac474d61a6960acc0b360dbc, State: Initialized, Role: FOLLOWER
I20240507 02:10:46.304731 16409 consensus_queue.cc:260] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a8b5ba9bf94f4aa922d3d5307406b6" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 38643 } } peers { permanent_uuid: "da19d23f7add4b50bf53330de753ce77" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 42387 } } peers { permanent_uuid: "0d6e5d3dac474d61a6960acc0b360dbc" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43457 } }
I20240507 02:10:46.304973 16409 ts_tablet_manager.cc:1398] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20240507 02:10:46.345654 16519 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:10:46.294988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:35343 (local address 127.31.131.65:33321)
0507 02:10:46.295125 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:46.295154 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0507 02:10:46.295570 (+ 416us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:10:46.345498 (+ 49928us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:35343: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":62,"threads_started":1}
W20240507 02:10:46.345898 16517 heartbeater.cc:644] Failed to heartbeat to 127.31.131.126:35343 (0 consecutive failures): Network error: Failed to ping master at 127.31.131.126:35343: Client connection negotiation failed: client connection to 127.31.131.126:35343: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240507 02:10:46.346244 16519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:46.346137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:35343 (local address 127.31.131.65:38789)
0507 02:10:46.346188 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:46.346222 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:35343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20240507 02:10:46.346596 16519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:46.346486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:35343 (local address 127.31.131.65:53235)
0507 02:10:46.346530 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:46.346576 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:35343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20240507 02:10:46.346696 16517 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20240507 02:10:46.377517 16521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:10:46.377674 16521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:10:46.377694 16521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:10:46.378940 16521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:10:46.378974 16521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:10:46.378996 16521 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.66
I20240507 02:10:46.380466 16521 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:43653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab
--principal=kudu/127.31.131.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.66:38643
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.131.66
--webserver_port=32811
--tserver_master_addrs=127.31.131.126:35343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16521
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.66
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:10:46.380618 16521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:10:46.380754 16521 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:10:46.384271 16530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:10:46.384271 16527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:10:46.384394 16528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:10:46.384408 16521 server_base.cc:1034] running on GCE node
I20240507 02:10:46.384634 16521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:10:46.384819 16521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:10:46.385944 16521 hybrid_clock.cc:648] HybridClock initialized: now 1715047846385933 us; error 24 us; skew 500 ppm
May 07 02:10:46 dist-test-slave-v0bg krb5kdc[15214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047846, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:10:46.388006 16521 init.cc:377] Logged in from keytab as kudu/127.31.131.66@KRBTEST.COM (short username kudu)
I20240507 02:10:46.388363 16521 webserver.cc:460] Webserver started at http://127.31.131.66:32811/ using document root <none> and password file <none>
I20240507 02:10:46.388523 16521 fs_manager.cc:359] Metadata directory not provided
I20240507 02:10:46.388562 16521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:10:46.389595 16521 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20240507 02:10:46.390041 16537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:10:46.390183 16521 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240507 02:10:46.390245 16521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-1/wal
uuid: "c5a8b5ba9bf94f4aa922d3d5307406b6"
format_stamp: "Formatted at 2024-05-07 02:10:05 on dist-test-slave-v0bg"
server_key: "da659846df383f31f89105bcdf6de9d0"
server_key_iv: "22baf740d3e4b3c8bcee96872b10d9c9"
server_key_version: "encryptionkey@0"
I20240507 02:10:46.390477 16521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:10:46.401342 16521 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:10:46.403062 16521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:10:46.403155 16521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:10:46.403311 16521 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:10:46.403550 16521 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:10:46.403579 16521 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:10:46.403597 16521 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:10:46.403620 16521 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:10:46.408489 16521 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.66:38643
I20240507 02:10:46.408543 16650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.66:38643 every 8 connection(s)
I20240507 02:10:46.408792 16521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
I20240507 02:10:46.409536 16653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:46.409258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:35343 (local address 127.31.131.66:38777)
0507 02:10:46.409371 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:46.409424 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:35343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":63,"threads_started":1}
W20240507 02:10:46.409686 16651 heartbeater.cc:644] Failed to heartbeat to 127.31.131.126:35343 (0 consecutive failures): Network error: Failed to ping master at 127.31.131.126:35343: Client connection negotiation failed: client connection to 127.31.131.126:35343: connect: Connection refused (error 111)
I20240507 02:10:46.409901 16653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:46.409828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:35343 (local address 127.31.131.66:39813)
0507 02:10:46.409864 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:46.409886 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:35343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20240507 02:10:46.410091 16653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:46.410036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:35343 (local address 127.31.131.66:40455)
0507 02:10:46.410064 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:46.410080 (+ 16us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:35343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20240507 02:10:46.410152 16651 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20240507 02:10:46.415890 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 16521
I20240507 02:10:46.416115 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.67:42387
--local_ip_for_outbound_sockets=127.31.131.67
--tserver_master_addrs=127.31.131.126:35343
--webserver_port=46439
--webserver_interface=127.31.131.67
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab
--principal=kudu/127.31.131.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:43653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:10:46.492169 16654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:10:46.492360 16654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:10:46.492385 16654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:10:46.493672 16654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:10:46.493716 16654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:10:46.493741 16654 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.67
I20240507 02:10:46.495299 16654 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:43653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab
--principal=kudu/127.31.131.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.67:42387
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.131.67
--webserver_port=46439
--tserver_master_addrs=127.31.131.126:35343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16654
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.67
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:10:46.495448 16654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:10:46.495587 16654 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:10:46.499377 16662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:10:46.499399 16659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:10:46.499516 16660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:10:46.499491 16654 server_base.cc:1034] running on GCE node
I20240507 02:10:46.499699 16654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:10:46.499886 16654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:10:46.501013 16654 hybrid_clock.cc:648] HybridClock initialized: now 1715047846500999 us; error 26 us; skew 500 ppm
May 07 02:10:46 dist-test-slave-v0bg krb5kdc[15214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047846, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:10:46.503064 16654 init.cc:377] Logged in from keytab as kudu/127.31.131.67@KRBTEST.COM (short username kudu)
I20240507 02:10:46.503417 16654 webserver.cc:460] Webserver started at http://127.31.131.67:46439/ using document root <none> and password file <none>
I20240507 02:10:46.503576 16654 fs_manager.cc:359] Metadata directory not provided
I20240507 02:10:46.503615 16654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:10:46.504637 16654 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20240507 02:10:46.505069 16669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:10:46.505213 16654 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240507 02:10:46.505276 16654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-2/wal
uuid: "da19d23f7add4b50bf53330de753ce77"
format_stamp: "Formatted at 2024-05-07 02:10:05 on dist-test-slave-v0bg"
server_key: "0bad5911b238d5d777b5e092a8e2d14b"
server_key_iv: "ddb757c23556d4f449a3d2b79f605e33"
server_key_version: "encryptionkey@0"
I20240507 02:10:46.505498 16654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:10:46.525424 16654 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:10:46.527192 16654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:10:46.527287 16654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:10:46.527444 16654 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:10:46.527685 16654 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:10:46.527714 16654 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:10:46.527740 16654 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:10:46.527761 16654 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:10:46.532577 16654 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.67:42387
I20240507 02:10:46.532631 16782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.67:42387 every 8 connection(s)
I20240507 02:10:46.532867 16654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
I20240507 02:10:46.533619 16785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:46.533367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:35343 (local address 127.31.131.67:42067)
0507 02:10:46.533473 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:46.533521 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:35343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79,"thread_start_us":51,"threads_started":1}
W20240507 02:10:46.533790 16783 heartbeater.cc:644] Failed to heartbeat to 127.31.131.126:35343 (0 consecutive failures): Network error: Failed to ping master at 127.31.131.126:35343: Client connection negotiation failed: client connection to 127.31.131.126:35343: connect: Connection refused (error 111)
I20240507 02:10:46.534003 16785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:46.533931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:35343 (local address 127.31.131.67:57907)
0507 02:10:46.533965 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:46.533987 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:35343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20240507 02:10:46.534217 16785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:46.534158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:35343 (local address 127.31.131.67:58191)
0507 02:10:46.534188 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:46.534205 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:35343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20240507 02:10:46.534297 16783 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20240507 02:10:46.539731 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 16654
I20240507 02:10:47.347183 16787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:47.346968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:35343 (local address 127.31.131.65:40225)
0507 02:10:47.347086 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:47.347122 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:35343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":63,"threads_started":1}
I20240507 02:10:47.410641 16788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:47.410421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:35343 (local address 127.31.131.66:54701)
0507 02:10:47.410526 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:47.410594 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:35343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":64,"threads_started":1}
I20240507 02:10:47.534832 16789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:47.534622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:35343 (local address 127.31.131.67:54261)
0507 02:10:47.534759 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:47.534784 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:35343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":67,"threads_started":1}
I20240507 02:10:47.577919 16790 raft_consensus.cc:482] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 02:10:47.578056 16790 raft_consensus.cc:504] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a8b5ba9bf94f4aa922d3d5307406b6" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 38643 } } peers { permanent_uuid: "da19d23f7add4b50bf53330de753ce77" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 42387 } } peers { permanent_uuid: "0d6e5d3dac474d61a6960acc0b360dbc" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43457 } }
I20240507 02:10:47.578342 16790 leader_election.cc:288] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5a8b5ba9bf94f4aa922d3d5307406b6 (127.31.131.66:38643), da19d23f7add4b50bf53330de753ce77 (127.31.131.67:42387)
May 07 02:10:47 dist-test-slave-v0bg krb5kdc[15214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047846, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.65@KRBTEST.COM for kudu/127.31.131.67@KRBTEST.COM
May 07 02:10:47 dist-test-slave-v0bg krb5kdc[15214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047846, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.65@KRBTEST.COM for kudu/127.31.131.66@KRBTEST.COM
I20240507 02:10:47.587749 16793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:47.578594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:46619 (local address 127.31.131.67:42387)
0507 02:10:47.578736 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0507 02:10:47.578754 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0507 02:10:47.578775 (+ 21us) server_negotiation.cc:407] Connection header received
0507 02:10:47.579284 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:10:47.579287 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:10:47.579343 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:10:47.579501 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:10:47.580565 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:10:47.581751 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:10:47.582452 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:10:47.582710 (+ 258us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:10:47.584673 (+ 1963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:10:47.584683 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:10:47.584683 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:10:47.584704 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:10:47.586463 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:10:47.586831 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:10:47.586834 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:10:47.586834 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:10:47.586890 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:10:47.587056 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:10:47.587058 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:10:47.587058 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:10:47.587237 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:10:47.587309 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0507 02:10:47.587552 (+ 243us) server_negotiation.cc:299] Negotiation successful
0507 02:10:47.587652 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":64,"threads_started":1}
I20240507 02:10:47.587762 16791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:47.578466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.67:42387 (local address 127.31.131.65:46619)
0507 02:10:47.578619 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:47.578627 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0507 02:10:47.578669 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:10:47.579550 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:10:47.579552 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:10:47.580073 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:10:47.580464 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:10:47.580471 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:10:47.581839 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:10:47.581840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:10:47.582390 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:10:47.582391 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:10:47.582503 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:10:47.583200 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:10:47.583204 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:10:47.584558 (+ 1354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:10:47.586595 (+ 2037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:10:47.586600 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:10:47.586603 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:10:47.586770 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:10:47.586948 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:10:47.586948 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:10:47.586948 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:10:47.586990 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:10:47.587298 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:10:47.587303 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:10:47.587397 (+ 94us) client_negotiation.cc:769] Sending connection context
0507 02:10:47.587479 (+ 82us) client_negotiation.cc:240] Negotiation successful
0507 02:10:47.587627 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":10,"thread_start_us":45,"threads_started":1}
I20240507 02:10:47.587795 16792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:47.578603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:55903 (local address 127.31.131.66:38643)
0507 02:10:47.578736 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0507 02:10:47.578754 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0507 02:10:47.578775 (+ 21us) server_negotiation.cc:407] Connection header received
0507 02:10:47.579284 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:10:47.579288 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:10:47.579343 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:10:47.579501 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:10:47.580565 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:10:47.581752 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:10:47.582452 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:10:47.582710 (+ 258us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:10:47.585010 (+ 2300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:10:47.585017 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:10:47.585018 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:10:47.585032 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:10:47.586719 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:10:47.586997 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:10:47.586999 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:10:47.587000 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:10:47.587057 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:10:47.587186 (+ 129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:10:47.587187 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:10:47.587187 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:10:47.587353 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:10:47.587421 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0507 02:10:47.587586 (+ 165us) server_negotiation.cc:299] Negotiation successful
0507 02:10:47.587696 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":53,"threads_started":1}
I20240507 02:10:47.588477 16787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:47.578465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.66:38643 (local address 127.31.131.65:55903)
0507 02:10:47.578513 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:47.578523 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0507 02:10:47.578596 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:10:47.579549 (+ 953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:10:47.579552 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:10:47.580089 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:10:47.580464 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:10:47.580471 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:10:47.581845 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:10:47.581846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:10:47.582381 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:10:47.582383 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:10:47.582505 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:10:47.583138 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:10:47.583153 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:10:47.584927 (+ 1774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:10:47.586784 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:10:47.586785 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:10:47.586785 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:10:47.586941 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:10:47.587109 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:10:47.587109 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:10:47.587110 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:10:47.587137 (+ 27us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:10:47.587408 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:10:47.587409 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:10:47.587450 (+ 41us) client_negotiation.cc:769] Sending connection context
0507 02:10:47.587516 (+ 66us) client_negotiation.cc:240] Negotiation successful
0507 02:10:47.587698 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":86}
W20240507 02:10:47.588514 16405 leader_election.cc:341] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer da19d23f7add4b50bf53330de753ce77 (127.31.131.67:42387): Not found: Tablet not found: 1aca4532ebbe437d9839a2dd92e667bc
W20240507 02:10:47.589038 16406 leader_election.cc:341] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c5a8b5ba9bf94f4aa922d3d5307406b6 (127.31.131.66:38643): Not found: Tablet not found: 1aca4532ebbe437d9839a2dd92e667bc
I20240507 02:10:47.589107 16406 leader_election.cc:302] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d6e5d3dac474d61a6960acc0b360dbc; no voters: c5a8b5ba9bf94f4aa922d3d5307406b6, da19d23f7add4b50bf53330de753ce77
I20240507 02:10:47.589212 16790 raft_consensus.cc:2726] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 02:10:48.347837 16794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:48.347612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:35343 (local address 127.31.131.65:39305)
0507 02:10:48.347741 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:48.347785 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:35343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":61,"threads_started":1}
I20240507 02:10:48.411166 16795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:48.410993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:35343 (local address 127.31.131.66:46985)
0507 02:10:48.411102 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:48.411127 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:35343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":61,"threads_started":1}
I20240507 02:10:48.535346 16796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:48.535178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:35343 (local address 127.31.131.67:44091)
0507 02:10:48.535286 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:48.535316 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:35343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":53,"threads_started":1}
I20240507 02:10:49.348476 16797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:49.348282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:35343 (local address 127.31.131.65:50691)
0507 02:10:49.348396 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:49.348429 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:35343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":66,"threads_started":1}
I20240507 02:10:49.411737 16798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:49.411550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:35343 (local address 127.31.131.66:45177)
0507 02:10:49.411666 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:49.411699 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:35343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":66,"threads_started":1}
I20240507 02:10:49.512431 16799 raft_consensus.cc:482] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 02:10:49.512531 16799 raft_consensus.cc:504] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a8b5ba9bf94f4aa922d3d5307406b6" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 38643 } } peers { permanent_uuid: "da19d23f7add4b50bf53330de753ce77" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 42387 } } peers { permanent_uuid: "0d6e5d3dac474d61a6960acc0b360dbc" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43457 } }
I20240507 02:10:49.512696 16799 leader_election.cc:288] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5a8b5ba9bf94f4aa922d3d5307406b6 (127.31.131.66:38643), da19d23f7add4b50bf53330de753ce77 (127.31.131.67:42387)
W20240507 02:10:49.513062 16405 leader_election.cc:341] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer da19d23f7add4b50bf53330de753ce77 (127.31.131.67:42387): Not found: Tablet not found: 1aca4532ebbe437d9839a2dd92e667bc
W20240507 02:10:49.513191 16406 leader_election.cc:341] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c5a8b5ba9bf94f4aa922d3d5307406b6 (127.31.131.66:38643): Not found: Tablet not found: 1aca4532ebbe437d9839a2dd92e667bc
I20240507 02:10:49.513237 16406 leader_election.cc:302] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d6e5d3dac474d61a6960acc0b360dbc; no voters: c5a8b5ba9bf94f4aa922d3d5307406b6, da19d23f7add4b50bf53330de753ce77
I20240507 02:10:49.513299 16799 raft_consensus.cc:2726] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 02:10:49.535930 16800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:49.535732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:35343 (local address 127.31.131.67:59537)
0507 02:10:49.535842 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:49.535887 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:35343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":58,"threads_started":1}
I20240507 02:10:50.349153 16801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:50.348921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:35343 (local address 127.31.131.65:52639)
0507 02:10:50.349062 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:50.349103 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:35343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":70,"threads_started":1}
I20240507 02:10:50.412348 16802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:50.412139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:35343 (local address 127.31.131.66:40821)
0507 02:10:50.412263 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:50.412302 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:35343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":63,"threads_started":1}
I20240507 02:10:50.536471 16803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:50.536291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:35343 (local address 127.31.131.67:46343)
0507 02:10:50.536400 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:50.536432 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:35343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":62,"threads_started":1}
I20240507 02:10:51.349819 16804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:51.349607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:35343 (local address 127.31.131.65:56251)
0507 02:10:51.349741 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:51.349770 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:35343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":68,"threads_started":1}
I20240507 02:10:51.412943 16805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:51.412742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:35343 (local address 127.31.131.66:41685)
0507 02:10:51.412874 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:51.412904 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:35343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":78,"threads_started":1}
I20240507 02:10:51.537060 16806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:51.536872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:35343 (local address 127.31.131.67:58207)
0507 02:10:51.536987 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:51.537017 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:35343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":59,"threads_started":1}
I20240507 02:10:52.091984 16807 raft_consensus.cc:482] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 02:10:52.092092 16807 raft_consensus.cc:504] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a8b5ba9bf94f4aa922d3d5307406b6" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 38643 } } peers { permanent_uuid: "da19d23f7add4b50bf53330de753ce77" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 42387 } } peers { permanent_uuid: "0d6e5d3dac474d61a6960acc0b360dbc" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43457 } }
I20240507 02:10:52.092285 16807 leader_election.cc:288] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5a8b5ba9bf94f4aa922d3d5307406b6 (127.31.131.66:38643), da19d23f7add4b50bf53330de753ce77 (127.31.131.67:42387)
W20240507 02:10:52.092664 16405 leader_election.cc:341] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer da19d23f7add4b50bf53330de753ce77 (127.31.131.67:42387): Not found: Tablet not found: 1aca4532ebbe437d9839a2dd92e667bc
W20240507 02:10:52.092751 16406 leader_election.cc:341] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c5a8b5ba9bf94f4aa922d3d5307406b6 (127.31.131.66:38643): Not found: Tablet not found: 1aca4532ebbe437d9839a2dd92e667bc
I20240507 02:10:52.092797 16406 leader_election.cc:302] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d6e5d3dac474d61a6960acc0b360dbc; no voters: c5a8b5ba9bf94f4aa922d3d5307406b6, da19d23f7add4b50bf53330de753ce77
I20240507 02:10:52.092854 16807 raft_consensus.cc:2726] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 02:10:52.350446 16808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:52.350234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:35343 (local address 127.31.131.65:37743)
0507 02:10:52.350368 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:52.350399 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:35343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":79,"threads_started":1}
I20240507 02:10:52.413511 16809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:52.413314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:35343 (local address 127.31.131.66:50827)
0507 02:10:52.413431 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:52.413462 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:35343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":59,"threads_started":1}
I20240507 02:10:52.537617 16810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:52.537430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:35343 (local address 127.31.131.67:33839)
0507 02:10:52.537542 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:52.537577 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:35343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":59,"threads_started":1}
I20240507 02:10:53.351058 16811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:53.350841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:35343 (local address 127.31.131.65:44299)
0507 02:10:53.350975 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:53.351010 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:35343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":71,"threads_started":1}
I20240507 02:10:53.414108 16812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:53.413894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:35343 (local address 127.31.131.66:59975)
0507 02:10:53.414026 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:53.414057 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:35343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":74,"threads_started":1}
I20240507 02:10:53.538255 16813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:53.538032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:35343 (local address 127.31.131.67:36253)
0507 02:10:53.538178 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:53.538214 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:35343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":91,"threads_started":1}
I20240507 02:10:54.351666 16814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:54.351443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:35343 (local address 127.31.131.65:39457)
0507 02:10:54.351569 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:54.351618 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:35343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":76,"threads_started":1}
I20240507 02:10:54.414700 16815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:54.414489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:35343 (local address 127.31.131.66:42935)
0507 02:10:54.414625 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:54.414660 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:35343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":79,"threads_started":1}
I20240507 02:10:54.538872 16816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:54.538678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:35343 (local address 127.31.131.67:38529)
0507 02:10:54.538794 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:54.538827 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:35343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":62,"threads_started":1}
I20240507 02:10:55.352319 16817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:55.352095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:35343 (local address 127.31.131.65:35715)
0507 02:10:55.352234 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:55.352269 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:35343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":72,"threads_started":1}
I20240507 02:10:55.415246 16818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:55.415061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:35343 (local address 127.31.131.66:39951)
0507 02:10:55.415173 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:55.415205 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:35343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":63,"threads_started":1}
I20240507 02:10:55.484791 16819 raft_consensus.cc:482] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 02:10:55.484882 16819 raft_consensus.cc:504] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a8b5ba9bf94f4aa922d3d5307406b6" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 38643 } } peers { permanent_uuid: "da19d23f7add4b50bf53330de753ce77" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 42387 } } peers { permanent_uuid: "0d6e5d3dac474d61a6960acc0b360dbc" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43457 } }
I20240507 02:10:55.485038 16819 leader_election.cc:288] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5a8b5ba9bf94f4aa922d3d5307406b6 (127.31.131.66:38643), da19d23f7add4b50bf53330de753ce77 (127.31.131.67:42387)
W20240507 02:10:55.485368 16406 leader_election.cc:341] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c5a8b5ba9bf94f4aa922d3d5307406b6 (127.31.131.66:38643): Not found: Tablet not found: 1aca4532ebbe437d9839a2dd92e667bc
W20240507 02:10:55.485437 16405 leader_election.cc:341] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer da19d23f7add4b50bf53330de753ce77 (127.31.131.67:42387): Not found: Tablet not found: 1aca4532ebbe437d9839a2dd92e667bc
I20240507 02:10:55.485472 16405 leader_election.cc:302] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d6e5d3dac474d61a6960acc0b360dbc; no voters: c5a8b5ba9bf94f4aa922d3d5307406b6, da19d23f7add4b50bf53330de753ce77
I20240507 02:10:55.485538 16819 raft_consensus.cc:2726] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 02:10:55.539440 16820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:55.539253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:35343 (local address 127.31.131.67:43797)
0507 02:10:55.539363 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:55.539396 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:35343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":60,"threads_started":1}
I20240507 02:10:56.352926 16821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:56.352714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:35343 (local address 127.31.131.65:43189)
0507 02:10:56.352838 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:56.352873 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:35343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":59,"threads_started":1}
I20240507 02:10:56.415771 16822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:56.415586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:35343 (local address 127.31.131.66:34033)
0507 02:10:56.415699 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:56.415731 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:35343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":55,"threads_started":1}
I20240507 02:10:56.539961 16823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:56.539774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:35343 (local address 127.31.131.67:47809)
0507 02:10:56.539892 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:56.539923 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:35343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":59,"threads_started":1}
I20240507 02:10:57.353503 16824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:57.353301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:35343 (local address 127.31.131.65:60235)
0507 02:10:57.353421 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:57.353454 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:35343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":68,"threads_started":1}
I20240507 02:10:57.416327 16825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:57.416138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:35343 (local address 127.31.131.66:59839)
0507 02:10:57.416256 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:57.416286 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:35343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":69,"threads_started":1}
I20240507 02:10:57.540522 16826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:57.540323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:35343 (local address 127.31.131.67:60869)
0507 02:10:57.540445 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:57.540475 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:35343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":71,"threads_started":1}
I20240507 02:10:58.354142 16827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:58.353929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:35343 (local address 127.31.131.65:59641)
0507 02:10:58.354057 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:58.354092 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:35343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":79,"threads_started":1}
I20240507 02:10:58.416934 16828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:58.416728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:35343 (local address 127.31.131.66:51851)
0507 02:10:58.416856 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:58.416890 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:35343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":70,"threads_started":1}
I20240507 02:10:58.541076 16829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:58.540881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:35343 (local address 127.31.131.67:46467)
0507 02:10:58.541002 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:58.541032 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:35343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":75,"threads_started":1}
I20240507 02:10:58.929770 16830 raft_consensus.cc:482] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 02:10:58.929862 16830 raft_consensus.cc:504] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a8b5ba9bf94f4aa922d3d5307406b6" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 38643 } } peers { permanent_uuid: "da19d23f7add4b50bf53330de753ce77" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 42387 } } peers { permanent_uuid: "0d6e5d3dac474d61a6960acc0b360dbc" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43457 } }
I20240507 02:10:58.930014 16830 leader_election.cc:288] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5a8b5ba9bf94f4aa922d3d5307406b6 (127.31.131.66:38643), da19d23f7add4b50bf53330de753ce77 (127.31.131.67:42387)
W20240507 02:10:58.930313 16406 leader_election.cc:341] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c5a8b5ba9bf94f4aa922d3d5307406b6 (127.31.131.66:38643): Not found: Tablet not found: 1aca4532ebbe437d9839a2dd92e667bc
W20240507 02:10:58.930397 16405 leader_election.cc:341] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer da19d23f7add4b50bf53330de753ce77 (127.31.131.67:42387): Not found: Tablet not found: 1aca4532ebbe437d9839a2dd92e667bc
I20240507 02:10:58.930452 16405 leader_election.cc:302] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d6e5d3dac474d61a6960acc0b360dbc; no voters: c5a8b5ba9bf94f4aa922d3d5307406b6, da19d23f7add4b50bf53330de753ce77
I20240507 02:10:58.930534 16830 raft_consensus.cc:2726] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 02:10:59.354774 16831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:59.354565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:35343 (local address 127.31.131.65:59501)
0507 02:10:59.354698 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:59.354726 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:35343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":83,"threads_started":1}
I20240507 02:10:59.417518 16832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:59.417326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:35343 (local address 127.31.131.66:50759)
0507 02:10:59.417449 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:59.417479 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:35343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":73,"threads_started":1}
I20240507 02:10:59.541615 16833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:10:59.541438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:35343 (local address 127.31.131.67:58185)
0507 02:10:59.541540 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0507 02:10:59.541571 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:35343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":56,"threads_started":1}
I20240507 02:11:00.355333 16834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:00.355157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:35343 (local address 127.31.131.65:39165)
0507 02:11:00.355260 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:00.355287 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:35343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":60,"threads_started":1}
I20240507 02:11:00.418057 16835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:00.417892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:35343 (local address 127.31.131.66:47379)
0507 02:11:00.417994 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:00.418019 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:35343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":59,"threads_started":1}
I20240507 02:11:00.542152 16836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:00.541966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:35343 (local address 127.31.131.67:47591)
0507 02:11:00.542079 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:00.542111 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:35343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":59,"threads_started":1}
I20240507 02:11:01.042419 16837 raft_consensus.cc:482] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 02:11:01.042516 16837 raft_consensus.cc:504] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a8b5ba9bf94f4aa922d3d5307406b6" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 38643 } } peers { permanent_uuid: "da19d23f7add4b50bf53330de753ce77" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 42387 } } peers { permanent_uuid: "0d6e5d3dac474d61a6960acc0b360dbc" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 43457 } }
I20240507 02:11:01.042701 16837 leader_election.cc:288] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5a8b5ba9bf94f4aa922d3d5307406b6 (127.31.131.66:38643), da19d23f7add4b50bf53330de753ce77 (127.31.131.67:42387)
W20240507 02:11:01.043009 16406 leader_election.cc:341] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c5a8b5ba9bf94f4aa922d3d5307406b6 (127.31.131.66:38643): Not found: Tablet not found: 1aca4532ebbe437d9839a2dd92e667bc
W20240507 02:11:01.043175 16405 leader_election.cc:341] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer da19d23f7add4b50bf53330de753ce77 (127.31.131.67:42387): Not found: Tablet not found: 1aca4532ebbe437d9839a2dd92e667bc
I20240507 02:11:01.043224 16405 leader_election.cc:302] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d6e5d3dac474d61a6960acc0b360dbc; no voters: c5a8b5ba9bf94f4aa922d3d5307406b6, da19d23f7add4b50bf53330de753ce77
I20240507 02:11:01.043306 16837 raft_consensus.cc:2726] T 1aca4532ebbe437d9839a2dd92e667bc P 0d6e5d3dac474d61a6960acc0b360dbc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 02:11:01.355888 16838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:01.355711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:35343 (local address 127.31.131.65:41663)
0507 02:11:01.355816 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:01.355843 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:35343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":60,"threads_started":1}
I20240507 02:11:01.418622 16839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:01.418414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:35343 (local address 127.31.131.66:59461)
0507 02:11:01.418524 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:01.418576 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:35343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":62,"threads_started":1}
I20240507 02:11:01.540843 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 16353
I20240507 02:11:01.542793 16840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:01.542621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:35343 (local address 127.31.131.67:60239)
0507 02:11:01.542726 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:01.542756 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.126:35343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":60,"threads_started":1}
I20240507 02:11:01.545104 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 16521
I20240507 02:11:01.548827 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 16654
2024-05-07T02:11:01Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (57835 ms)
[----------] 30 tests from SecurityITest (257118 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 07 02:11:01 dist-test-slave-v0bg krb5kdc[16843](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 07 02:11:01 dist-test-slave-v0bg krb5kdc[16843](info): set up 2 sockets
May 07 02:11:01 dist-test-slave-v0bg krb5kdc[16843](info): commencing operation
krb5kdc: starting...
W20240507 02:11:02.430898 32269 mini_kdc.cc:121] Time spent starting KDC: real 0.864s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/krb5kdc/test-user.keytab.
May 07 02:11:02 dist-test-slave-v0bg krb5kdc[16843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-07T02:11:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-07T02:11:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.131.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab.
Entry for principal kudu/127.31.131.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab.
I20240507 02:11:02.532935 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:35065
--webserver_interface=127.31.131.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab
--principal=kudu/127.31.131.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:36489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.131.126:35065
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:11:02.607281 16856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:11:02.607452 16856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:11:02.607470 16856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:11:02.608673 16856 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240507 02:11:02.608706 16856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:11:02.608718 16856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:11:02.608729 16856 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240507 02:11:02.608738 16856 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240507 02:11:02.610224 16856 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:36489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.131.126:35065
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab
--principal=kudu/127.31.131.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:35065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.131.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:11:02.610381 16856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:11:02.610532 16856 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:11:02.614328 16864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:02.614408 16856 server_base.cc:1034] running on GCE node
W20240507 02:11:02.614534 16861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:11:02.614650 16862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:02.614923 16856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:11:02.615149 16856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:11:02.616289 16856 hybrid_clock.cc:648] HybridClock initialized: now 1715047862616268 us; error 32 us; skew 500 ppm
May 07 02:11:02 dist-test-slave-v0bg krb5kdc[16843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047862, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:11:02.618496 16856 init.cc:377] Logged in from keytab as kudu/127.31.131.126@KRBTEST.COM (short username kudu)
I20240507 02:11:02.618880 16856 webserver.cc:460] Webserver started at http://127.31.131.126:44747/ using document root <none> and password file <none>
I20240507 02:11:02.619040 16856 fs_manager.cc:359] Metadata directory not provided
I20240507 02:11:02.619081 16856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:11:02.619189 16856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:11:02.620105 16856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/minicluster-data/master-0/data/instance:
uuid: "a7495c872219483cae510e34fc1f54dd"
format_stamp: "Formatted at 2024-05-07 02:11:02 on dist-test-slave-v0bg"
server_key: "7b81de5077bae5a40a849fec57c0b839"
server_key_iv: "348b306c1eb52e0568b0ab199c60820a"
server_key_version: "encryptionkey@0"
I20240507 02:11:02.620397 16856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/minicluster-data/master-0/wal/instance:
uuid: "a7495c872219483cae510e34fc1f54dd"
format_stamp: "Formatted at 2024-05-07 02:11:02 on dist-test-slave-v0bg"
server_key: "7b81de5077bae5a40a849fec57c0b839"
server_key_iv: "348b306c1eb52e0568b0ab199c60820a"
server_key_version: "encryptionkey@0"
I20240507 02:11:02.621491 16856 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20240507 02:11:02.622067 16871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:02.622215 16856 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:02.622293 16856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/minicluster-data/master-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/minicluster-data/master-0/wal
uuid: "a7495c872219483cae510e34fc1f54dd"
format_stamp: "Formatted at 2024-05-07 02:11:02 on dist-test-slave-v0bg"
server_key: "7b81de5077bae5a40a849fec57c0b839"
server_key_iv: "348b306c1eb52e0568b0ab199c60820a"
server_key_version: "encryptionkey@0"
I20240507 02:11:02.622352 16856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:11:02.664397 16856 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:11:02.665944 16856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:11:02.666054 16856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:11:02.669332 16856 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.126:35065
I20240507 02:11:02.669371 16923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.126:35065 every 8 connection(s)
I20240507 02:11:02.669658 16856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
I20240507 02:11:02.670320 16924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:11:02.672336 16924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7495c872219483cae510e34fc1f54dd: Bootstrap starting.
I20240507 02:11:02.672847 16924 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7495c872219483cae510e34fc1f54dd: Neither blocks nor log segments found. Creating new log.
I20240507 02:11:02.673084 16924 log.cc:824] T 00000000000000000000000000000000 P a7495c872219483cae510e34fc1f54dd: Log is configured to *not* fsync() on all Append() calls
I20240507 02:11:02.673652 16924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7495c872219483cae510e34fc1f54dd: No bootstrap required, opened a new log
I20240507 02:11:02.675012 16924 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7495c872219483cae510e34fc1f54dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7495c872219483cae510e34fc1f54dd" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 35065 } }
I20240507 02:11:02.675127 16924 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7495c872219483cae510e34fc1f54dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:11:02.675143 16924 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7495c872219483cae510e34fc1f54dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7495c872219483cae510e34fc1f54dd, State: Initialized, Role: FOLLOWER
I20240507 02:11:02.675259 16924 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7495c872219483cae510e34fc1f54dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7495c872219483cae510e34fc1f54dd" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 35065 } }
I20240507 02:11:02.675314 16924 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7495c872219483cae510e34fc1f54dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 02:11:02.675334 16924 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7495c872219483cae510e34fc1f54dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 02:11:02.675374 16924 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7495c872219483cae510e34fc1f54dd [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:11:02.675848 16924 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7495c872219483cae510e34fc1f54dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7495c872219483cae510e34fc1f54dd" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 35065 } }
I20240507 02:11:02.675932 16924 leader_election.cc:302] T 00000000000000000000000000000000 P a7495c872219483cae510e34fc1f54dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7495c872219483cae510e34fc1f54dd; no voters:
I20240507 02:11:02.676059 16924 leader_election.cc:288] T 00000000000000000000000000000000 P a7495c872219483cae510e34fc1f54dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20240507 02:11:02.676098 16927 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7495c872219483cae510e34fc1f54dd [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:11:02.676244 16927 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7495c872219483cae510e34fc1f54dd [term 1 LEADER]: Becoming Leader. State: Replica: a7495c872219483cae510e34fc1f54dd, State: Running, Role: LEADER
I20240507 02:11:02.676316 16927 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7495c872219483cae510e34fc1f54dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7495c872219483cae510e34fc1f54dd" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 35065 } }
I20240507 02:11:02.676561 16924 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7495c872219483cae510e34fc1f54dd [sys.catalog]: configured and running, proceeding with master startup.
I20240507 02:11:02.676714 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 16856
I20240507 02:11:02.676823 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/minicluster-data/master-0/wal/instance
I20240507 02:11:02.677021 32269 external_mini_cluster.cc:1428] Setting key 51abf47a5d90cf8e20aeb5c67dea9213
I20240507 02:11:02.677258 16929 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7495c872219483cae510e34fc1f54dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7495c872219483cae510e34fc1f54dd. Latest consensus state: current_term: 1 leader_uuid: "a7495c872219483cae510e34fc1f54dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7495c872219483cae510e34fc1f54dd" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 35065 } } }
I20240507 02:11:02.677332 16929 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7495c872219483cae510e34fc1f54dd [sys.catalog]: This master's current role is: LEADER
I20240507 02:11:02.677423 16928 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7495c872219483cae510e34fc1f54dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a7495c872219483cae510e34fc1f54dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7495c872219483cae510e34fc1f54dd" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 35065 } } }
I20240507 02:11:02.677485 16928 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7495c872219483cae510e34fc1f54dd [sys.catalog]: This master's current role is: LEADER
I20240507 02:11:02.677590 16937 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 02:11:02.678061 16937 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240507 02:11:02.679387 16937 catalog_manager.cc:1347] Generated new cluster ID: 8022a6c713144785aefbc65481f4f50e
I20240507 02:11:02.679433 16937 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
May 07 02:11:02 dist-test-slave-v0bg krb5kdc[16843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:11:02.686864 16942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:02.677592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47128 (local address 127.31.131.126:35065)
0507 02:11:02.677801 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:02.677812 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:02.678006 (+ 194us) server_negotiation.cc:407] Connection header received
0507 02:11:02.678178 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:02.678183 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:02.678230 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:02.678368 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:02.679249 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:02.680432 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:02.681159 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:02.681383 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:02.683770 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:02.683776 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:02.683776 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:11:02.683794 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:02.685576 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:02.685912 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:02.685915 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:02.685915 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:02.685960 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:02.686097 (+ 137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:02.686098 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:02.686098 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:02.686294 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:02.686414 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:02.686532 (+ 118us) server_negotiation.cc:299] Negotiation successful
0507 02:11:02.686670 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":63,"threads_started":1}
I20240507 02:11:02.712093 16937 catalog_manager.cc:1370] Generated new certificate authority record
I20240507 02:11:02.712591 16937 catalog_manager.cc:1504] Loading token signing keys...
I20240507 02:11:02.718312 16937 catalog_manager.cc:5857] T 00000000000000000000000000000000 P a7495c872219483cae510e34fc1f54dd: Generated new TSK 0
I20240507 02:11:02.718458 16937 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.131.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab.
Entry for principal kudu/127.31.131.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab.
I20240507 02:11:02.747066 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.65:0
--local_ip_for_outbound_sockets=127.31.131.65
--webserver_interface=127.31.131.65
--webserver_port=0
--tserver_master_addrs=127.31.131.126:35065
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab
--principal=kudu/127.31.131.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:36489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:11:02.820631 16951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:11:02.820787 16951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:11:02.820807 16951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:11:02.822048 16951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:11:02.822086 16951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:11:02.822113 16951 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.65
I20240507 02:11:02.823624 16951 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:36489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab
--principal=kudu/127.31.131.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.131.65
--webserver_port=0
--tserver_master_addrs=127.31.131.126:35065
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.65
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:11:02.823791 16951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:11:02.823947 16951 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240507 02:11:02.827556 16951 server_base.cc:1034] running on GCE node
W20240507 02:11:02.827584 16959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:11:02.827582 16957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:11:02.827740 16956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:02.827931 16951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:11:02.828115 16951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:11:02.829234 16951 hybrid_clock.cc:648] HybridClock initialized: now 1715047862829225 us; error 22 us; skew 500 ppm
May 07 02:11:02 dist-test-slave-v0bg krb5kdc[16843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047862, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:11:02.831339 16951 init.cc:377] Logged in from keytab as kudu/127.31.131.65@KRBTEST.COM (short username kudu)
I20240507 02:11:02.831707 16951 webserver.cc:460] Webserver started at http://127.31.131.65:44499/ using document root <none> and password file <none>
I20240507 02:11:02.831853 16951 fs_manager.cc:359] Metadata directory not provided
I20240507 02:11:02.831890 16951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:11:02.831985 16951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:11:02.832829 16951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/minicluster-data/ts-0/data/instance:
uuid: "3d21b03c963644919780993b7dba26ba"
format_stamp: "Formatted at 2024-05-07 02:11:02 on dist-test-slave-v0bg"
server_key: "a2a3b1f78719cea0b99f762e476f277f"
server_key_iv: "2e5b77f6baf768955d2a316dff651551"
server_key_version: "encryptionkey@0"
I20240507 02:11:02.833089 16951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/minicluster-data/ts-0/wal/instance:
uuid: "3d21b03c963644919780993b7dba26ba"
format_stamp: "Formatted at 2024-05-07 02:11:02 on dist-test-slave-v0bg"
server_key: "a2a3b1f78719cea0b99f762e476f277f"
server_key_iv: "2e5b77f6baf768955d2a316dff651551"
server_key_version: "encryptionkey@0"
I20240507 02:11:02.834061 16951 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240507 02:11:02.834640 16966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:02.834761 16951 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:02.834828 16951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/minicluster-data/ts-0/wal
uuid: "3d21b03c963644919780993b7dba26ba"
format_stamp: "Formatted at 2024-05-07 02:11:02 on dist-test-slave-v0bg"
server_key: "a2a3b1f78719cea0b99f762e476f277f"
server_key_iv: "2e5b77f6baf768955d2a316dff651551"
server_key_version: "encryptionkey@0"
I20240507 02:11:02.834883 16951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:11:02.845978 16951 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:11:02.847347 16951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:11:02.847441 16951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:11:02.847602 16951 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:11:02.847841 16951 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:11:02.847870 16951 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:02.847894 16951 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:11:02.847915 16951 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:02.852685 16951 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.65:35297
I20240507 02:11:02.852733 17079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.65:35297 every 8 connection(s)
I20240507 02:11:02.852958 16951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
May 07 02:11:02 dist-test-slave-v0bg krb5kdc[16843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047862, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.65@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:11:02.859642 16942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:02.853519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:43869 (local address 127.31.131.126:35065)
0507 02:11:02.853573 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:02.853574 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:02.853702 (+ 128us) server_negotiation.cc:407] Connection header received
0507 02:11:02.854101 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:02.854101 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:02.854112 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:02.854164 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:02.855039 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:02.855514 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:02.855976 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:02.856075 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:02.857703 (+ 1628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:02.857706 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:02.857707 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:11:02.857713 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:02.858798 (+ 1085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:02.859089 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:02.859090 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:02.859090 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:02.859105 (+ 15us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:02.859222 (+ 117us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:02.859222 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:02.859222 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:02.859297 (+ 75us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:02.859337 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:02.859509 (+ 172us) server_negotiation.cc:299] Negotiation successful
0507 02:11:02.859580 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20240507 02:11:02.859714 17082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:02.853469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:35065 (local address 127.31.131.65:43869)
0507 02:11:02.853606 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:02.853631 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0507 02:11:02.854015 (+ 384us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:11:02.854203 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:11:02.854206 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:11:02.854461 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:11:02.854965 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:02.854969 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:02.855572 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:11:02.855572 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:11:02.855927 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:02.855928 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:02.856015 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:02.856411 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:11:02.856422 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:11:02.857625 (+ 1203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:11:02.858866 (+ 1241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:11:02.858870 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:11:02.858872 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:11:02.859039 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:11:02.859144 (+ 105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:11:02.859145 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:11:02.859145 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:11:02.859187 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:11:02.859334 (+ 147us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:11:02.859336 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:11:02.859421 (+ 85us) client_negotiation.cc:769] Sending connection context
0507 02:11:02.859495 (+ 74us) client_negotiation.cc:240] Negotiation successful
0507 02:11:02.859589 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":76,"threads_started":1}
I20240507 02:11:02.860153 17080 heartbeater.cc:347] Connected to a master server at 127.31.131.126:35065
I20240507 02:11:02.860219 17080 heartbeater.cc:464] Registering TS with master...
I20240507 02:11:02.860376 17080 heartbeater.cc:511] Master 127.31.131.126:35065 requested a full tablet report, sending...
I20240507 02:11:02.860383 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 16951
I20240507 02:11:02.860474 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/minicluster-data/ts-0/wal/instance
I20240507 02:11:02.860646 32269 external_mini_cluster.cc:1428] Setting key 88899bddad33e48a93b55c046d450d55
I20240507 02:11:02.860875 16888 ts_manager.cc:194] Registered new tserver with Master: 3d21b03c963644919780993b7dba26ba (127.31.131.65:35297)
I20240507 02:11:02.861696 16888 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.131.65@KRBTEST.COM'} at 127.31.131.65:43869
WARNING: no policy specified for kudu/127.31.131.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab.
Entry for principal kudu/127.31.131.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab.
I20240507 02:11:02.881067 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.66:0
--local_ip_for_outbound_sockets=127.31.131.66
--webserver_interface=127.31.131.66
--webserver_port=0
--tserver_master_addrs=127.31.131.126:35065
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab
--principal=kudu/127.31.131.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:36489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:11:02.956423 17085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:11:02.956591 17085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:11:02.956612 17085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:11:02.957827 17085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:11:02.957862 17085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:11:02.957885 17085 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.66
I20240507 02:11:02.959365 17085 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:36489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab
--principal=kudu/127.31.131.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.131.66
--webserver_port=0
--tserver_master_addrs=127.31.131.126:35065
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17085
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.66
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:11:02.959523 17085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:11:02.959662 17085 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:11:02.963281 17090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:02.963284 17085 server_base.cc:1034] running on GCE node
W20240507 02:11:02.963353 17091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:11:02.963418 17093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:02.963583 17085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:11:02.963757 17085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:11:02.964883 17085 hybrid_clock.cc:648] HybridClock initialized: now 1715047862964869 us; error 30 us; skew 500 ppm
May 07 02:11:02 dist-test-slave-v0bg krb5kdc[16843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047862, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:11:02.966982 17085 init.cc:377] Logged in from keytab as kudu/127.31.131.66@KRBTEST.COM (short username kudu)
I20240507 02:11:02.967348 17085 webserver.cc:460] Webserver started at http://127.31.131.66:46245/ using document root <none> and password file <none>
I20240507 02:11:02.967499 17085 fs_manager.cc:359] Metadata directory not provided
I20240507 02:11:02.967543 17085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:11:02.967639 17085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:11:02.968516 17085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/minicluster-data/ts-1/data/instance:
uuid: "1190c27257e34028b8339699c03c952f"
format_stamp: "Formatted at 2024-05-07 02:11:02 on dist-test-slave-v0bg"
server_key: "b9b8c6939b6d6ee510fced5773edb39c"
server_key_iv: "4a194c350ebf15e0f451f118b3e95a3e"
server_key_version: "encryptionkey@0"
I20240507 02:11:02.968778 17085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/minicluster-data/ts-1/wal/instance:
uuid: "1190c27257e34028b8339699c03c952f"
format_stamp: "Formatted at 2024-05-07 02:11:02 on dist-test-slave-v0bg"
server_key: "b9b8c6939b6d6ee510fced5773edb39c"
server_key_iv: "4a194c350ebf15e0f451f118b3e95a3e"
server_key_version: "encryptionkey@0"
I20240507 02:11:02.969779 17085 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240507 02:11:02.970326 17100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:02.970465 17085 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:02.970543 17085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/minicluster-data/ts-1/wal
uuid: "1190c27257e34028b8339699c03c952f"
format_stamp: "Formatted at 2024-05-07 02:11:02 on dist-test-slave-v0bg"
server_key: "b9b8c6939b6d6ee510fced5773edb39c"
server_key_iv: "4a194c350ebf15e0f451f118b3e95a3e"
server_key_version: "encryptionkey@0"
I20240507 02:11:02.970619 17085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:11:02.988754 17085 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:11:02.990201 17085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:11:02.990303 17085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:11:02.990485 17085 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:11:02.990803 17085 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:11:02.990833 17085 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:02.990860 17085 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:11:02.990882 17085 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:02.995728 17085 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.66:43117
I20240507 02:11:02.995779 17213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.66:43117 every 8 connection(s)
I20240507 02:11:02.996026 17085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
May 07 02:11:02 dist-test-slave-v0bg krb5kdc[16843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047862, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.66@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:11:03.002928 16942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:02.996634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.66:50337 (local address 127.31.131.126:35065)
0507 02:11:02.996695 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:02.996697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:02.996827 (+ 130us) server_negotiation.cc:407] Connection header received
0507 02:11:02.997257 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:02.997257 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:02.997266 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:02.997316 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:02.998179 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:02.998670 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:02.999147 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:02.999246 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:03.000895 (+ 1649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:03.000899 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:03.000899 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:11:03.000906 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:03.002084 (+ 1178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:03.002351 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:03.002352 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:03.002352 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:03.002368 (+ 16us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:03.002480 (+ 112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:03.002480 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:03.002480 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:03.002579 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:03.002627 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:03.002803 (+ 176us) server_negotiation.cc:299] Negotiation successful
0507 02:11:03.002864 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20240507 02:11:03.003021 17216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:02.996594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:35065 (local address 127.31.131.66:50337)
0507 02:11:02.996727 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:02.996754 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0507 02:11:02.997180 (+ 426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:11:02.997365 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:11:02.997368 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:11:02.997627 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:11:02.998114 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:02.998117 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:02.998727 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:11:02.998727 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:11:02.999093 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:02.999095 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:02.999189 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:02.999612 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:11:02.999624 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:11:03.000816 (+ 1192us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:11:03.002144 (+ 1328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:11:03.002147 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:11:03.002149 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:11:03.002302 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:11:03.002408 (+ 106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:11:03.002408 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:11:03.002408 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:11:03.002450 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:11:03.002624 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:11:03.002626 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:11:03.002716 (+ 90us) client_negotiation.cc:769] Sending connection context
0507 02:11:03.002795 (+ 79us) client_negotiation.cc:240] Negotiation successful
0507 02:11:03.002884 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":69,"threads_started":1}
I20240507 02:11:03.003443 17214 heartbeater.cc:347] Connected to a master server at 127.31.131.126:35065
I20240507 02:11:03.003523 17214 heartbeater.cc:464] Registering TS with master...
I20240507 02:11:03.003679 17214 heartbeater.cc:511] Master 127.31.131.126:35065 requested a full tablet report, sending...
I20240507 02:11:03.004009 16888 ts_manager.cc:194] Registered new tserver with Master: 1190c27257e34028b8339699c03c952f (127.31.131.66:43117)
I20240507 02:11:03.004357 16888 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.131.66@KRBTEST.COM'} at 127.31.131.66:50337
I20240507 02:11:03.004671 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 17085
I20240507 02:11:03.004760 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/minicluster-data/ts-1/wal/instance
I20240507 02:11:03.004913 32269 external_mini_cluster.cc:1428] Setting key 9392ecb9b14744cf3ad6c77d59c799b6
WARNING: no policy specified for kudu/127.31.131.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab.
Entry for principal kudu/127.31.131.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab.
I20240507 02:11:03.025107 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.67:0
--local_ip_for_outbound_sockets=127.31.131.67
--webserver_interface=127.31.131.67
--webserver_port=0
--tserver_master_addrs=127.31.131.126:35065
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab
--principal=kudu/127.31.131.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:36489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:11:03.098836 17219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:11:03.098990 17219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:11:03.099010 17219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:11:03.100206 17219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:11:03.100241 17219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:11:03.100263 17219 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.67
I20240507 02:11:03.101671 17219 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:36489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab
--principal=kudu/127.31.131.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.131.67
--webserver_port=0
--tserver_master_addrs=127.31.131.126:35065
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17219
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.67
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:11:03.101819 17219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:11:03.101948 17219 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:11:03.105530 17225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:11:03.105600 17224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:03.105695 17219 server_base.cc:1034] running on GCE node
W20240507 02:11:03.105742 17227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:03.105917 17219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:11:03.106096 17219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:11:03.107223 17219 hybrid_clock.cc:648] HybridClock initialized: now 1715047863107207 us; error 35 us; skew 500 ppm
May 07 02:11:03 dist-test-slave-v0bg krb5kdc[16843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047863, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:11:03.109365 17219 init.cc:377] Logged in from keytab as kudu/127.31.131.67@KRBTEST.COM (short username kudu)
I20240507 02:11:03.109696 17219 webserver.cc:460] Webserver started at http://127.31.131.67:46557/ using document root <none> and password file <none>
I20240507 02:11:03.109838 17219 fs_manager.cc:359] Metadata directory not provided
I20240507 02:11:03.109869 17219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:11:03.109941 17219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:11:03.110827 17219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/minicluster-data/ts-2/data/instance:
uuid: "b20fe15b06df4d60a5ba826ed9adf50d"
format_stamp: "Formatted at 2024-05-07 02:11:03 on dist-test-slave-v0bg"
server_key: "c9b95ae243c947f99f8d9b39314b4f35"
server_key_iv: "4672db5db928de9463d0d9eab2c39533"
server_key_version: "encryptionkey@0"
I20240507 02:11:03.111131 17219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/minicluster-data/ts-2/wal/instance:
uuid: "b20fe15b06df4d60a5ba826ed9adf50d"
format_stamp: "Formatted at 2024-05-07 02:11:03 on dist-test-slave-v0bg"
server_key: "c9b95ae243c947f99f8d9b39314b4f35"
server_key_iv: "4672db5db928de9463d0d9eab2c39533"
server_key_version: "encryptionkey@0"
I20240507 02:11:03.112133 17219 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240507 02:11:03.112668 17234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:03.112805 17219 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240507 02:11:03.112865 17219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/minicluster-data/ts-2/wal
uuid: "b20fe15b06df4d60a5ba826ed9adf50d"
format_stamp: "Formatted at 2024-05-07 02:11:03 on dist-test-slave-v0bg"
server_key: "c9b95ae243c947f99f8d9b39314b4f35"
server_key_iv: "4672db5db928de9463d0d9eab2c39533"
server_key_version: "encryptionkey@0"
I20240507 02:11:03.112926 17219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:11:03.128736 17219 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:11:03.130095 17219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:11:03.130183 17219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:11:03.130352 17219 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:11:03.130640 17219 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:11:03.130671 17219 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:03.130690 17219 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:11:03.130712 17219 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:03.135461 17219 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.67:45123
I20240507 02:11:03.135515 17347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.67:45123 every 8 connection(s)
I20240507 02:11:03.135730 17219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
I20240507 02:11:03.138566 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 17219
I20240507 02:11:03.138653 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/minicluster-data/ts-2/wal/instance
I20240507 02:11:03.138810 32269 external_mini_cluster.cc:1428] Setting key e39370c869e36dd3b5a7b1131b61651f
May 07 02:11:03 dist-test-slave-v0bg krb5kdc[16843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047863, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.67@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:11:03.143077 16942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:03.136295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.67:59927 (local address 127.31.131.126:35065)
0507 02:11:03.136350 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:03.136352 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:03.136491 (+ 139us) server_negotiation.cc:407] Connection header received
0507 02:11:03.136909 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:03.136909 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:03.136918 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:03.136970 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:03.137866 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:03.138339 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:03.138876 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:03.139020 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:03.140805 (+ 1785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:03.140808 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:03.140809 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:11:03.140815 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:03.141998 (+ 1183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:03.142304 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:03.142304 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:03.142305 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:03.142322 (+ 17us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:03.142447 (+ 125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:03.142448 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:03.142448 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:03.142537 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:03.142627 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:03.142897 (+ 270us) server_negotiation.cc:299] Negotiation successful
0507 02:11:03.142982 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20240507 02:11:03.143167 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:03.136261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:35065 (local address 127.31.131.67:59927)
0507 02:11:03.136393 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:03.136418 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0507 02:11:03.136830 (+ 412us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:11:03.137015 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:11:03.137018 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:11:03.137270 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:11:03.137797 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:03.137801 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:03.138400 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:11:03.138401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:11:03.138801 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:03.138803 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:03.138919 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:03.139362 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:11:03.139381 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:11:03.140733 (+ 1352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:11:03.142070 (+ 1337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:11:03.142074 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:11:03.142077 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:11:03.142243 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:11:03.142366 (+ 123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:11:03.142367 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:11:03.142367 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:11:03.142408 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:11:03.142626 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:11:03.142632 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:11:03.142767 (+ 135us) client_negotiation.cc:769] Sending connection context
0507 02:11:03.142869 (+ 102us) client_negotiation.cc:240] Negotiation successful
0507 02:11:03.143000 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":65,"threads_started":1}
I20240507 02:11:03.143574 17348 heartbeater.cc:347] Connected to a master server at 127.31.131.126:35065
I20240507 02:11:03.143651 17348 heartbeater.cc:464] Registering TS with master...
I20240507 02:11:03.143813 17348 heartbeater.cc:511] Master 127.31.131.126:35065 requested a full tablet report, sending...
I20240507 02:11:03.144141 16888 ts_manager.cc:194] Registered new tserver with Master: b20fe15b06df4d60a5ba826ed9adf50d (127.31.131.67:45123)
I20240507 02:11:03.144502 16888 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.131.67@KRBTEST.COM'} at 127.31.131.67:59927
I20240507 02:11:03.150789 32269 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240507 02:11:03.156668 16942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:03.152008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47140 (local address 127.31.131.126:35065)
0507 02:11:03.152061 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:03.152063 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:03.152140 (+ 77us) server_negotiation.cc:407] Connection header received
0507 02:11:03.152178 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:03.152178 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:03.152186 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:03.152232 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:03.152811 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:03.153257 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:03.153764 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:03.153875 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:03.154886 (+ 1011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:03.154888 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:03.154889 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:11:03.154894 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:03.155939 (+ 1045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:03.156184 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:03.156184 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:03.156185 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:03.156199 (+ 14us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:03.156303 (+ 104us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:03.156303 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:03.156303 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:03.156392 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:03.156458 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:03.156553 (+ 95us) server_negotiation.cc:299] Negotiation successful
0507 02:11:03.156609 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20240507 02:11:03.157985 16888 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47140:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 02:11:03.159413 16888 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 02:11:03.164067 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:03.161225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.65:35297 (local address 127.0.0.1:43058)
0507 02:11:03.161388 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:03.161410 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0507 02:11:03.161471 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:11:03.161747 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:11:03.161749 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:11:03.161752 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:11:03.161936 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:03.161937 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:03.163206 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:11:03.163207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:11:03.163848 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:03.163850 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:03.163911 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:03.163933 (+ 22us) client_negotiation.cc:769] Sending connection context
0507 02:11:03.163969 (+ 36us) client_negotiation.cc:240] Negotiation successful
0507 02:11:03.164002 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":62,"threads_started":1}
I20240507 02:11:03.164151 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:03.161225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.66:43117 (local address 127.0.0.1:55832)
0507 02:11:03.161585 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:03.161595 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0507 02:11:03.161642 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:11:03.161852 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:11:03.161853 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:11:03.161855 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:11:03.162058 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:03.162060 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:03.163350 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:11:03.163351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:11:03.163945 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:03.163947 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:03.164019 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:03.164022 (+ 3us) client_negotiation.cc:769] Sending connection context
0507 02:11:03.164050 (+ 28us) client_negotiation.cc:240] Negotiation successful
0507 02:11:03.164087 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"mutex_wait_us":3,"thread_start_us":58,"threads_started":1}
I20240507 02:11:03.164738 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:03.161347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55832 (local address 127.31.131.66:43117)
0507 02:11:03.161679 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:03.161692 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:03.161704 (+ 12us) server_negotiation.cc:407] Connection header received
0507 02:11:03.161743 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:03.161745 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:03.161761 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:03.161843 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:11:03.162184 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:03.163279 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:03.164061 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:03.164549 (+ 488us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:03.164602 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:03.164634 (+ 32us) server_negotiation.cc:299] Negotiation successful
0507 02:11:03.164676 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":80,"threads_started":1}
I20240507 02:11:03.164934 17358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:03.161302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43058 (local address 127.31.131.65:35297)
0507 02:11:03.161503 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:03.161510 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:03.161520 (+ 10us) server_negotiation.cc:407] Connection header received
0507 02:11:03.161559 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:03.161561 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:03.161578 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:03.161663 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:11:03.162010 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:03.163123 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:03.163949 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:03.164750 (+ 801us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:03.164793 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:03.164824 (+ 31us) server_negotiation.cc:299] Negotiation successful
0507 02:11:03.164859 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":94,"threads_started":1}
I20240507 02:11:03.165197 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:03.161433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.67:45123 (local address 127.0.0.1:35966)
0507 02:11:03.161946 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:03.161955 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0507 02:11:03.162010 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:11:03.162258 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:11:03.162259 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:11:03.162260 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:11:03.162440 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:03.162441 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:03.164203 (+ 1762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:11:03.164204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:11:03.165012 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:03.165015 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:03.165080 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:03.165083 (+ 3us) client_negotiation.cc:769] Sending connection context
0507 02:11:03.165110 (+ 27us) client_negotiation.cc:240] Negotiation successful
0507 02:11:03.165145 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":86,"threads_started":1}
I20240507 02:11:03.165256 17148 tablet_service.cc:1467] Processing CreateTablet for tablet 08fb68740e024eccbfab9b0c5e2c160d (DEFAULT_TABLE table=test-table [id=516f71f6d7d5493daea16a135d64a3c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:11:03.165436 17014 tablet_service.cc:1467] Processing CreateTablet for tablet 08fb68740e024eccbfab9b0c5e2c160d (DEFAULT_TABLE table=test-table [id=516f71f6d7d5493daea16a135d64a3c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:11:03.165571 17148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08fb68740e024eccbfab9b0c5e2c160d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:11:03.165676 17014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08fb68740e024eccbfab9b0c5e2c160d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:11:03.165891 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:03.161635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35966 (local address 127.31.131.67:45123)
0507 02:11:03.161814 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:03.161820 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:03.162068 (+ 248us) server_negotiation.cc:407] Connection header received
0507 02:11:03.162105 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:03.162107 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:03.162123 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:03.162192 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:11:03.162524 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:03.163264 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:03.165346 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:03.165767 (+ 421us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:03.165794 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:03.165818 (+ 24us) server_negotiation.cc:299] Negotiation successful
0507 02:11:03.165842 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":98,"threads_started":1}
I20240507 02:11:03.166301 17282 tablet_service.cc:1467] Processing CreateTablet for tablet 08fb68740e024eccbfab9b0c5e2c160d (DEFAULT_TABLE table=test-table [id=516f71f6d7d5493daea16a135d64a3c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:11:03.166523 17282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08fb68740e024eccbfab9b0c5e2c160d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:11:03.168653 17362 tablet_bootstrap.cc:492] T 08fb68740e024eccbfab9b0c5e2c160d P 3d21b03c963644919780993b7dba26ba: Bootstrap starting.
I20240507 02:11:03.168875 17363 tablet_bootstrap.cc:492] T 08fb68740e024eccbfab9b0c5e2c160d P 1190c27257e34028b8339699c03c952f: Bootstrap starting.
I20240507 02:11:03.168916 17364 tablet_bootstrap.cc:492] T 08fb68740e024eccbfab9b0c5e2c160d P b20fe15b06df4d60a5ba826ed9adf50d: Bootstrap starting.
I20240507 02:11:03.169317 17362 tablet_bootstrap.cc:654] T 08fb68740e024eccbfab9b0c5e2c160d P 3d21b03c963644919780993b7dba26ba: Neither blocks nor log segments found. Creating new log.
I20240507 02:11:03.169423 17363 tablet_bootstrap.cc:654] T 08fb68740e024eccbfab9b0c5e2c160d P 1190c27257e34028b8339699c03c952f: Neither blocks nor log segments found. Creating new log.
I20240507 02:11:03.169423 17364 tablet_bootstrap.cc:654] T 08fb68740e024eccbfab9b0c5e2c160d P b20fe15b06df4d60a5ba826ed9adf50d: Neither blocks nor log segments found. Creating new log.
I20240507 02:11:03.169723 17363 log.cc:824] T 08fb68740e024eccbfab9b0c5e2c160d P 1190c27257e34028b8339699c03c952f: Log is configured to *not* fsync() on all Append() calls
I20240507 02:11:03.169723 17364 log.cc:824] T 08fb68740e024eccbfab9b0c5e2c160d P b20fe15b06df4d60a5ba826ed9adf50d: Log is configured to *not* fsync() on all Append() calls
I20240507 02:11:03.169858 17362 log.cc:824] T 08fb68740e024eccbfab9b0c5e2c160d P 3d21b03c963644919780993b7dba26ba: Log is configured to *not* fsync() on all Append() calls
I20240507 02:11:03.170336 17364 tablet_bootstrap.cc:492] T 08fb68740e024eccbfab9b0c5e2c160d P b20fe15b06df4d60a5ba826ed9adf50d: No bootstrap required, opened a new log
I20240507 02:11:03.170392 17364 ts_tablet_manager.cc:1367] T 08fb68740e024eccbfab9b0c5e2c160d P b20fe15b06df4d60a5ba826ed9adf50d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240507 02:11:03.170394 17363 tablet_bootstrap.cc:492] T 08fb68740e024eccbfab9b0c5e2c160d P 1190c27257e34028b8339699c03c952f: No bootstrap required, opened a new log
I20240507 02:11:03.170449 17363 ts_tablet_manager.cc:1367] T 08fb68740e024eccbfab9b0c5e2c160d P 1190c27257e34028b8339699c03c952f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240507 02:11:03.170832 17362 tablet_bootstrap.cc:492] T 08fb68740e024eccbfab9b0c5e2c160d P 3d21b03c963644919780993b7dba26ba: No bootstrap required, opened a new log
I20240507 02:11:03.170902 17362 ts_tablet_manager.cc:1367] T 08fb68740e024eccbfab9b0c5e2c160d P 3d21b03c963644919780993b7dba26ba: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240507 02:11:03.172019 17364 raft_consensus.cc:348] T 08fb68740e024eccbfab9b0c5e2c160d P b20fe15b06df4d60a5ba826ed9adf50d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d21b03c963644919780993b7dba26ba" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35297 } } peers { permanent_uuid: "1190c27257e34028b8339699c03c952f" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 43117 } } peers { permanent_uuid: "b20fe15b06df4d60a5ba826ed9adf50d" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 45123 } }
I20240507 02:11:03.172129 17364 raft_consensus.cc:374] T 08fb68740e024eccbfab9b0c5e2c160d P b20fe15b06df4d60a5ba826ed9adf50d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:11:03.172154 17364 raft_consensus.cc:729] T 08fb68740e024eccbfab9b0c5e2c160d P b20fe15b06df4d60a5ba826ed9adf50d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b20fe15b06df4d60a5ba826ed9adf50d, State: Initialized, Role: FOLLOWER
I20240507 02:11:03.172250 17364 consensus_queue.cc:260] T 08fb68740e024eccbfab9b0c5e2c160d P b20fe15b06df4d60a5ba826ed9adf50d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d21b03c963644919780993b7dba26ba" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35297 } } peers { permanent_uuid: "1190c27257e34028b8339699c03c952f" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 43117 } } peers { permanent_uuid: "b20fe15b06df4d60a5ba826ed9adf50d" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 45123 } }
I20240507 02:11:03.172473 17364 ts_tablet_manager.cc:1398] T 08fb68740e024eccbfab9b0c5e2c160d P b20fe15b06df4d60a5ba826ed9adf50d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240507 02:11:03.172510 17348 heartbeater.cc:503] Master 127.31.131.126:35065 was elected leader, sending a full tablet report...
I20240507 02:11:03.172516 17363 raft_consensus.cc:348] T 08fb68740e024eccbfab9b0c5e2c160d P 1190c27257e34028b8339699c03c952f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d21b03c963644919780993b7dba26ba" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35297 } } peers { permanent_uuid: "1190c27257e34028b8339699c03c952f" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 43117 } } peers { permanent_uuid: "b20fe15b06df4d60a5ba826ed9adf50d" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 45123 } }
I20240507 02:11:03.172631 17363 raft_consensus.cc:374] T 08fb68740e024eccbfab9b0c5e2c160d P 1190c27257e34028b8339699c03c952f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:11:03.172653 17363 raft_consensus.cc:729] T 08fb68740e024eccbfab9b0c5e2c160d P 1190c27257e34028b8339699c03c952f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1190c27257e34028b8339699c03c952f, State: Initialized, Role: FOLLOWER
I20240507 02:11:03.172735 17363 consensus_queue.cc:260] T 08fb68740e024eccbfab9b0c5e2c160d P 1190c27257e34028b8339699c03c952f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d21b03c963644919780993b7dba26ba" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35297 } } peers { permanent_uuid: "1190c27257e34028b8339699c03c952f" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 43117 } } peers { permanent_uuid: "b20fe15b06df4d60a5ba826ed9adf50d" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 45123 } }
I20240507 02:11:03.172966 17363 ts_tablet_manager.cc:1398] T 08fb68740e024eccbfab9b0c5e2c160d P 1190c27257e34028b8339699c03c952f: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240507 02:11:03.172971 17362 raft_consensus.cc:348] T 08fb68740e024eccbfab9b0c5e2c160d P 3d21b03c963644919780993b7dba26ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d21b03c963644919780993b7dba26ba" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35297 } } peers { permanent_uuid: "1190c27257e34028b8339699c03c952f" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 43117 } } peers { permanent_uuid: "b20fe15b06df4d60a5ba826ed9adf50d" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 45123 } }
I20240507 02:11:03.173034 17214 heartbeater.cc:503] Master 127.31.131.126:35065 was elected leader, sending a full tablet report...
I20240507 02:11:03.173070 17362 raft_consensus.cc:374] T 08fb68740e024eccbfab9b0c5e2c160d P 3d21b03c963644919780993b7dba26ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:11:03.173094 17362 raft_consensus.cc:729] T 08fb68740e024eccbfab9b0c5e2c160d P 3d21b03c963644919780993b7dba26ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d21b03c963644919780993b7dba26ba, State: Initialized, Role: FOLLOWER
I20240507 02:11:03.173166 17362 consensus_queue.cc:260] T 08fb68740e024eccbfab9b0c5e2c160d P 3d21b03c963644919780993b7dba26ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d21b03c963644919780993b7dba26ba" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35297 } } peers { permanent_uuid: "1190c27257e34028b8339699c03c952f" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 43117 } } peers { permanent_uuid: "b20fe15b06df4d60a5ba826ed9adf50d" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 45123 } }
I20240507 02:11:03.173364 17362 ts_tablet_manager.cc:1398] T 08fb68740e024eccbfab9b0c5e2c160d P 3d21b03c963644919780993b7dba26ba: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240507 02:11:03.173457 17080 heartbeater.cc:503] Master 127.31.131.126:35065 was elected leader, sending a full tablet report...
W20240507 02:11:03.246469 17215 tablet.cc:2369] T 08fb68740e024eccbfab9b0c5e2c160d P 1190c27257e34028b8339699c03c952f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240507 02:11:03.353535 17081 tablet.cc:2369] T 08fb68740e024eccbfab9b0c5e2c160d P 3d21b03c963644919780993b7dba26ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240507 02:11:03.386188 17349 tablet.cc:2369] T 08fb68740e024eccbfab9b0c5e2c160d P b20fe15b06df4d60a5ba826ed9adf50d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240507 02:11:03.390962 17370 raft_consensus.cc:482] T 08fb68740e024eccbfab9b0c5e2c160d P 3d21b03c963644919780993b7dba26ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 02:11:03.391067 17370 raft_consensus.cc:504] T 08fb68740e024eccbfab9b0c5e2c160d P 3d21b03c963644919780993b7dba26ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d21b03c963644919780993b7dba26ba" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35297 } } peers { permanent_uuid: "1190c27257e34028b8339699c03c952f" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 43117 } } peers { permanent_uuid: "b20fe15b06df4d60a5ba826ed9adf50d" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 45123 } }
I20240507 02:11:03.391394 17370 leader_election.cc:288] T 08fb68740e024eccbfab9b0c5e2c160d P 3d21b03c963644919780993b7dba26ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1190c27257e34028b8339699c03c952f (127.31.131.66:43117), b20fe15b06df4d60a5ba826ed9adf50d (127.31.131.67:45123)
I20240507 02:11:03.393508 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:03.391558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.66:43117 (local address 127.31.131.65:45707)
0507 02:11:03.391689 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:03.391697 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0507 02:11:03.391762 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:11:03.391888 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:11:03.391889 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:11:03.391891 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:11:03.392117 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:03.392119 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:03.392713 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:11:03.392714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:11:03.393261 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:03.393262 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:03.393315 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:03.393318 (+ 3us) client_negotiation.cc:769] Sending connection context
0507 02:11:03.393343 (+ 25us) client_negotiation.cc:240] Negotiation successful
0507 02:11:03.393378 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":67,"threads_started":1}
I20240507 02:11:03.393694 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:03.391556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.67:45123 (local address 127.31.131.65:53857)
0507 02:11:03.391674 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:03.391685 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0507 02:11:03.391748 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:11:03.391879 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:11:03.391880 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:11:03.391882 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:11:03.392116 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:03.392118 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:03.392741 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:11:03.392742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:11:03.393290 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:03.393292 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:03.393358 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:03.393361 (+ 3us) client_negotiation.cc:769] Sending connection context
0507 02:11:03.393387 (+ 26us) client_negotiation.cc:240] Negotiation successful
0507 02:11:03.393421 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":66,"threads_started":1}
I20240507 02:11:03.393889 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:03.391621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:53857 (local address 127.31.131.67:45123)
0507 02:11:03.391663 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:03.391665 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:03.391750 (+ 85us) server_negotiation.cc:407] Connection header received
0507 02:11:03.391805 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:03.391806 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:03.391816 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:03.391860 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:11:03.392183 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:03.392651 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:03.393376 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:03.393785 (+ 409us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:03.393793 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:03.393817 (+ 24us) server_negotiation.cc:299] Negotiation successful
0507 02:11:03.393847 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20240507 02:11:03.393893 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:03.391616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:45707 (local address 127.31.131.66:43117)
0507 02:11:03.391677 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:03.391679 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:03.391765 (+ 86us) server_negotiation.cc:407] Connection header received
0507 02:11:03.391827 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:03.391828 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:03.391837 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:03.391880 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:11:03.392185 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:03.392646 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:03.393341 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:03.393770 (+ 429us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:03.393779 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:03.393803 (+ 24us) server_negotiation.cc:299] Negotiation successful
0507 02:11:03.393836 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20240507 02:11:03.394131 17302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08fb68740e024eccbfab9b0c5e2c160d" candidate_uuid: "3d21b03c963644919780993b7dba26ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b20fe15b06df4d60a5ba826ed9adf50d" is_pre_election: true
I20240507 02:11:03.394135 17168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08fb68740e024eccbfab9b0c5e2c160d" candidate_uuid: "3d21b03c963644919780993b7dba26ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1190c27257e34028b8339699c03c952f" is_pre_election: true
I20240507 02:11:03.394258 17302 raft_consensus.cc:2454] T 08fb68740e024eccbfab9b0c5e2c160d P b20fe15b06df4d60a5ba826ed9adf50d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d21b03c963644919780993b7dba26ba in term 0.
I20240507 02:11:03.394264 17168 raft_consensus.cc:2454] T 08fb68740e024eccbfab9b0c5e2c160d P 1190c27257e34028b8339699c03c952f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d21b03c963644919780993b7dba26ba in term 0.
I20240507 02:11:03.394431 16967 leader_election.cc:302] T 08fb68740e024eccbfab9b0c5e2c160d P 3d21b03c963644919780993b7dba26ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1190c27257e34028b8339699c03c952f, 3d21b03c963644919780993b7dba26ba; no voters:
I20240507 02:11:03.394575 17370 raft_consensus.cc:2781] T 08fb68740e024eccbfab9b0c5e2c160d P 3d21b03c963644919780993b7dba26ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 02:11:03.394630 17370 raft_consensus.cc:482] T 08fb68740e024eccbfab9b0c5e2c160d P 3d21b03c963644919780993b7dba26ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 02:11:03.394651 17370 raft_consensus.cc:3037] T 08fb68740e024eccbfab9b0c5e2c160d P 3d21b03c963644919780993b7dba26ba [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:11:03.395251 17370 raft_consensus.cc:504] T 08fb68740e024eccbfab9b0c5e2c160d P 3d21b03c963644919780993b7dba26ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d21b03c963644919780993b7dba26ba" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35297 } } peers { permanent_uuid: "1190c27257e34028b8339699c03c952f" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 43117 } } peers { permanent_uuid: "b20fe15b06df4d60a5ba826ed9adf50d" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 45123 } }
I20240507 02:11:03.395426 17370 leader_election.cc:288] T 08fb68740e024eccbfab9b0c5e2c160d P 3d21b03c963644919780993b7dba26ba [CANDIDATE]: Term 1 election: Requested vote from peers 1190c27257e34028b8339699c03c952f (127.31.131.66:43117), b20fe15b06df4d60a5ba826ed9adf50d (127.31.131.67:45123)
I20240507 02:11:03.395509 17302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08fb68740e024eccbfab9b0c5e2c160d" candidate_uuid: "3d21b03c963644919780993b7dba26ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b20fe15b06df4d60a5ba826ed9adf50d"
I20240507 02:11:03.395576 17302 raft_consensus.cc:3037] T 08fb68740e024eccbfab9b0c5e2c160d P b20fe15b06df4d60a5ba826ed9adf50d [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:11:03.396200 17302 raft_consensus.cc:2454] T 08fb68740e024eccbfab9b0c5e2c160d P b20fe15b06df4d60a5ba826ed9adf50d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d21b03c963644919780993b7dba26ba in term 1.
I20240507 02:11:03.396293 17168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08fb68740e024eccbfab9b0c5e2c160d" candidate_uuid: "3d21b03c963644919780993b7dba26ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1190c27257e34028b8339699c03c952f"
I20240507 02:11:03.396363 17168 raft_consensus.cc:3037] T 08fb68740e024eccbfab9b0c5e2c160d P 1190c27257e34028b8339699c03c952f [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:11:03.396382 16968 leader_election.cc:302] T 08fb68740e024eccbfab9b0c5e2c160d P 3d21b03c963644919780993b7dba26ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d21b03c963644919780993b7dba26ba, b20fe15b06df4d60a5ba826ed9adf50d; no voters:
I20240507 02:11:03.396471 17370 raft_consensus.cc:2781] T 08fb68740e024eccbfab9b0c5e2c160d P 3d21b03c963644919780993b7dba26ba [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:11:03.396720 17370 raft_consensus.cc:686] T 08fb68740e024eccbfab9b0c5e2c160d P 3d21b03c963644919780993b7dba26ba [term 1 LEADER]: Becoming Leader. State: Replica: 3d21b03c963644919780993b7dba26ba, State: Running, Role: LEADER
I20240507 02:11:03.396888 17370 consensus_queue.cc:237] T 08fb68740e024eccbfab9b0c5e2c160d P 3d21b03c963644919780993b7dba26ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d21b03c963644919780993b7dba26ba" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35297 } } peers { permanent_uuid: "1190c27257e34028b8339699c03c952f" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 43117 } } peers { permanent_uuid: "b20fe15b06df4d60a5ba826ed9adf50d" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 45123 } }
I20240507 02:11:03.397058 17168 raft_consensus.cc:2454] T 08fb68740e024eccbfab9b0c5e2c160d P 1190c27257e34028b8339699c03c952f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d21b03c963644919780993b7dba26ba in term 1.
I20240507 02:11:03.397704 16888 catalog_manager.cc:5484] T 08fb68740e024eccbfab9b0c5e2c160d P 3d21b03c963644919780993b7dba26ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d21b03c963644919780993b7dba26ba (127.31.131.65). New cstate: current_term: 1 leader_uuid: "3d21b03c963644919780993b7dba26ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d21b03c963644919780993b7dba26ba" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 35297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1190c27257e34028b8339699c03c952f" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 43117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b20fe15b06df4d60a5ba826ed9adf50d" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 45123 } health_report { overall_health: UNKNOWN } } }
I20240507 02:11:03.444564 17358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:03.442863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43066 (local address 127.31.131.65:35297)
0507 02:11:03.442916 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:03.442917 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:03.442925 (+ 8us) server_negotiation.cc:407] Connection header received
0507 02:11:03.442959 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:03.442959 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:03.442975 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:03.443038 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0507 02:11:03.443250 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:03.443706 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:03.444203 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:03.444308 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:03.444339 (+ 31us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0507 02:11:03.444418 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0507 02:11:03.444470 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:03.444487 (+ 17us) server_negotiation.cc:299] Negotiation successful
0507 02:11:03.444502 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20240507 02:11:03.446089 17302 raft_consensus.cc:1261] T 08fb68740e024eccbfab9b0c5e2c160d P b20fe15b06df4d60a5ba826ed9adf50d [term 1 FOLLOWER]: Refusing update from remote peer 3d21b03c963644919780993b7dba26ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 02:11:03.446091 17168 raft_consensus.cc:1261] T 08fb68740e024eccbfab9b0c5e2c160d P 1190c27257e34028b8339699c03c952f [term 1 FOLLOWER]: Refusing update from remote peer 3d21b03c963644919780993b7dba26ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 02:11:03.446345 17370 consensus_queue.cc:1035] T 08fb68740e024eccbfab9b0c5e2c160d P 3d21b03c963644919780993b7dba26ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "b20fe15b06df4d60a5ba826ed9adf50d" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 45123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 02:11:03.446475 17373 consensus_queue.cc:1035] T 08fb68740e024eccbfab9b0c5e2c160d P 3d21b03c963644919780993b7dba26ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "1190c27257e34028b8339699c03c952f" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 02:11:03.448892 17378 mvcc.cc:205] Tried to move back new op lower bound from 7024836048672944128 to 7024836048474316800. Current Snapshot: MvccSnapshot[applied={T|T < 7024836048672944128 or (T in {7024836048672944128})}]
I20240507 02:11:03.448998 17377 mvcc.cc:205] Tried to move back new op lower bound from 7024836048672944128 to 7024836048474316800. Current Snapshot: MvccSnapshot[applied={T|T < 7024836048672944128}]
I20240507 02:11:03.450467 16888 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47140:
table { table_name: "test-table" } modify_external_catalogs: true
I20240507 02:11:03.450577 16888 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47140:
table { table_name: "test-table" } modify_external_catalogs: true
I20240507 02:11:03.451220 16888 catalog_manager.cc:5771] T 00000000000000000000000000000000 P a7495c872219483cae510e34fc1f54dd: Sending DeleteTablet for 3 replicas of tablet 08fb68740e024eccbfab9b0c5e2c160d
I20240507 02:11:03.451545 17014 tablet_service.cc:1514] Processing DeleteTablet for tablet 08fb68740e024eccbfab9b0c5e2c160d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:11:03 UTC) from {username='kudu', principal='kudu/127.31.131.126@KRBTEST.COM'} at 127.0.0.1:43058
I20240507 02:11:03.451542 17148 tablet_service.cc:1514] Processing DeleteTablet for tablet 08fb68740e024eccbfab9b0c5e2c160d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:11:03 UTC) from {username='kudu', principal='kudu/127.31.131.126@KRBTEST.COM'} at 127.0.0.1:55832
I20240507 02:11:03.451745 17387 tablet_replica.cc:330] T 08fb68740e024eccbfab9b0c5e2c160d P 1190c27257e34028b8339699c03c952f: stopping tablet replica
I20240507 02:11:03.451828 17387 raft_consensus.cc:2229] T 08fb68740e024eccbfab9b0c5e2c160d P 1190c27257e34028b8339699c03c952f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 02:11:03.451907 17387 raft_consensus.cc:2258] T 08fb68740e024eccbfab9b0c5e2c160d P 1190c27257e34028b8339699c03c952f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 02:11:03.452157 17386 tablet_replica.cc:330] T 08fb68740e024eccbfab9b0c5e2c160d P 3d21b03c963644919780993b7dba26ba: stopping tablet replica
I20240507 02:11:03.452271 17387 ts_tablet_manager.cc:1875] T 08fb68740e024eccbfab9b0c5e2c160d P 1190c27257e34028b8339699c03c952f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 02:11:03.452267 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 16951
I20240507 02:11:03.452280 17386 raft_consensus.cc:2229] T 08fb68740e024eccbfab9b0c5e2c160d P 3d21b03c963644919780993b7dba26ba [term 1 LEADER]: Raft consensus shutting down.
I20240507 02:11:03.452433 17386 raft_consensus.cc:2258] T 08fb68740e024eccbfab9b0c5e2c160d P 3d21b03c963644919780993b7dba26ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 02:11:03.453807 17387 ts_tablet_manager.cc:1888] T 08fb68740e024eccbfab9b0c5e2c160d P 1190c27257e34028b8339699c03c952f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240507 02:11:03.453868 17387 log.cc:1196] T 08fb68740e024eccbfab9b0c5e2c160d P 1190c27257e34028b8339699c03c952f: Deleting WAL directory at /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/minicluster-data/ts-1/wal/wals/08fb68740e024eccbfab9b0c5e2c160d
I20240507 02:11:03.454080 17387 ts_tablet_manager.cc:1909] T 08fb68740e024eccbfab9b0c5e2c160d P 1190c27257e34028b8339699c03c952f: Deleting consensus metadata
I20240507 02:11:03.454401 16872 catalog_manager.cc:4830] TS 1190c27257e34028b8339699c03c952f (127.31.131.66:43117): tablet 08fb68740e024eccbfab9b0c5e2c160d (table test-table [id=516f71f6d7d5493daea16a135d64a3c9]) successfully deleted
I20240507 02:11:03.456696 17282 tablet_service.cc:1514] Processing DeleteTablet for tablet 08fb68740e024eccbfab9b0c5e2c160d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:11:03 UTC) from {username='kudu', principal='kudu/127.31.131.126@KRBTEST.COM'} at 127.0.0.1:35966
I20240507 02:11:03.456876 17388 tablet_replica.cc:330] T 08fb68740e024eccbfab9b0c5e2c160d P b20fe15b06df4d60a5ba826ed9adf50d: stopping tablet replica
I20240507 02:11:03.456950 17388 raft_consensus.cc:2229] T 08fb68740e024eccbfab9b0c5e2c160d P b20fe15b06df4d60a5ba826ed9adf50d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 02:11:03.457024 17388 raft_consensus.cc:2258] T 08fb68740e024eccbfab9b0c5e2c160d P b20fe15b06df4d60a5ba826ed9adf50d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 02:11:03.457340 17388 ts_tablet_manager.cc:1875] T 08fb68740e024eccbfab9b0c5e2c160d P b20fe15b06df4d60a5ba826ed9adf50d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 02:11:03.458691 17388 ts_tablet_manager.cc:1888] T 08fb68740e024eccbfab9b0c5e2c160d P b20fe15b06df4d60a5ba826ed9adf50d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240507 02:11:03.458753 17388 log.cc:1196] T 08fb68740e024eccbfab9b0c5e2c160d P b20fe15b06df4d60a5ba826ed9adf50d: Deleting WAL directory at /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1715047604443657-32269-0/minicluster-data/ts-2/wal/wals/08fb68740e024eccbfab9b0c5e2c160d
I20240507 02:11:03.458998 17388 ts_tablet_manager.cc:1909] T 08fb68740e024eccbfab9b0c5e2c160d P b20fe15b06df4d60a5ba826ed9adf50d: Deleting consensus metadata
I20240507 02:11:03.459350 16873 catalog_manager.cc:4830] TS b20fe15b06df4d60a5ba826ed9adf50d (127.31.131.67:45123): tablet 08fb68740e024eccbfab9b0c5e2c160d (table test-table [id=516f71f6d7d5493daea16a135d64a3c9]) successfully deleted
W20240507 02:11:03.460371 16873 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.131.65:35297 (error 108)
I20240507 02:11:03.460574 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 17085
I20240507 02:11:03.461073 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:03.460692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.65:35297 (local address 127.0.0.1:43080)
0507 02:11:03.460999 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:03.461035 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.65:35297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":288}
W20240507 02:11:03.461650 16873 catalog_manager.cc:4573] TS 3d21b03c963644919780993b7dba26ba (127.31.131.65:35297): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 08fb68740e024eccbfab9b0c5e2c160d: Network error: Client connection negotiation failed: client connection to 127.31.131.65:35297: connect: Connection refused (error 111)
I20240507 02:11:03.464367 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 17219
I20240507 02:11:03.468411 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 16856
2024-05-07T02:11:03Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1919 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2024-05-07T02:11:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-07T02:11:03Z Disabled control of system clock
I20240507 02:11:03.491837 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715047604443657-32269-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715047604443657-32269-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715047604443657-32269-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:33953
--webserver_interface=127.31.131.126
--webserver_port=0
--builtin_ntp_servers=127.31.131.84:39431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.131.126:33953
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20240507 02:11:03.567654 17393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:11:03.567797 17393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:11:03.567816 17393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:11:03.568989 17393 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240507 02:11:03.569021 17393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:11:03.569032 17393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:11:03.569042 17393 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240507 02:11:03.569064 17393 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240507 02:11:03.570376 17393 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:39431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715047604443657-32269-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715047604443657-32269-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.131.126:33953
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:33953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.131.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17393
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715047604443657-32269-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:11:03.570529 17393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:11:03.570719 17393 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:11:03.574113 17401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:11:03.574155 17398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:11:03.574307 17399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:03.574399 17393 server_base.cc:1034] running on GCE node
I20240507 02:11:03.574622 17393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:11:03.574852 17393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:11:03.575973 17393 hybrid_clock.cc:648] HybridClock initialized: now 1715047863575960 us; error 27 us; skew 500 ppm
I20240507 02:11:03.576918 17393 webserver.cc:460] Webserver started at http://127.31.131.126:41679/ using document root <none> and password file <none>
I20240507 02:11:03.577081 17393 fs_manager.cc:359] Metadata directory not provided
I20240507 02:11:03.577121 17393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:11:03.577215 17393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:11:03.578097 17393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715047604443657-32269-0/minicluster-data/master-0/data/instance:
uuid: "aaea554e49f94f188e3a94678100870b"
format_stamp: "Formatted at 2024-05-07 02:11:03 on dist-test-slave-v0bg"
server_key: "c47c1307d95d4a087616c2ec0743497d"
server_key_iv: "fc8ddd82e061908eb581d154dc591807"
server_key_version: "encryptionkey@0"
I20240507 02:11:03.578353 17393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715047604443657-32269-0/minicluster-data/master-0/wal/instance:
uuid: "aaea554e49f94f188e3a94678100870b"
format_stamp: "Formatted at 2024-05-07 02:11:03 on dist-test-slave-v0bg"
server_key: "c47c1307d95d4a087616c2ec0743497d"
server_key_iv: "fc8ddd82e061908eb581d154dc591807"
server_key_version: "encryptionkey@0"
I20240507 02:11:03.579367 17393 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240507 02:11:03.579921 17407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:03.580057 17393 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:03.580123 17393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715047604443657-32269-0/minicluster-data/master-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715047604443657-32269-0/minicluster-data/master-0/wal
uuid: "aaea554e49f94f188e3a94678100870b"
format_stamp: "Formatted at 2024-05-07 02:11:03 on dist-test-slave-v0bg"
server_key: "c47c1307d95d4a087616c2ec0743497d"
server_key_iv: "fc8ddd82e061908eb581d154dc591807"
server_key_version: "encryptionkey@0"
I20240507 02:11:03.580176 17393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715047604443657-32269-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715047604443657-32269-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715047604443657-32269-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:11:03.588308 17393 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:11:03.588507 17393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:11:03.588603 17393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:11:03.591656 17393 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.126:33953
I20240507 02:11:03.591683 17459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.126:33953 every 8 connection(s)
I20240507 02:11:03.591922 17393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
I20240507 02:11:03.592396 17460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:11:03.594630 17460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aaea554e49f94f188e3a94678100870b: Bootstrap starting.
I20240507 02:11:03.594794 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 17393
I20240507 02:11:03.594897 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715047604443657-32269-0/minicluster-data/master-0/wal/instance
I20240507 02:11:03.595064 32269 external_mini_cluster.cc:1428] Setting key ee56392df37760225c3ce8c62d696357
I20240507 02:11:03.595274 17460 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aaea554e49f94f188e3a94678100870b: Neither blocks nor log segments found. Creating new log.
I20240507 02:11:03.595552 17460 log.cc:824] T 00000000000000000000000000000000 P aaea554e49f94f188e3a94678100870b: Log is configured to *not* fsync() on all Append() calls
I20240507 02:11:03.596144 17460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aaea554e49f94f188e3a94678100870b: No bootstrap required, opened a new log
I20240507 02:11:03.597255 17460 raft_consensus.cc:348] T 00000000000000000000000000000000 P aaea554e49f94f188e3a94678100870b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaea554e49f94f188e3a94678100870b" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 33953 } }
I20240507 02:11:03.597358 17460 raft_consensus.cc:374] T 00000000000000000000000000000000 P aaea554e49f94f188e3a94678100870b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:11:03.597373 17460 raft_consensus.cc:729] T 00000000000000000000000000000000 P aaea554e49f94f188e3a94678100870b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaea554e49f94f188e3a94678100870b, State: Initialized, Role: FOLLOWER
I20240507 02:11:03.597451 17460 consensus_queue.cc:260] T 00000000000000000000000000000000 P aaea554e49f94f188e3a94678100870b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaea554e49f94f188e3a94678100870b" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 33953 } }
I20240507 02:11:03.597509 17460 raft_consensus.cc:388] T 00000000000000000000000000000000 P aaea554e49f94f188e3a94678100870b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 02:11:03.597529 17460 raft_consensus.cc:482] T 00000000000000000000000000000000 P aaea554e49f94f188e3a94678100870b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 02:11:03.597568 17460 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aaea554e49f94f188e3a94678100870b [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:11:03.598018 17460 raft_consensus.cc:504] T 00000000000000000000000000000000 P aaea554e49f94f188e3a94678100870b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaea554e49f94f188e3a94678100870b" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 33953 } }
I20240507 02:11:03.598102 17460 leader_election.cc:302] T 00000000000000000000000000000000 P aaea554e49f94f188e3a94678100870b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aaea554e49f94f188e3a94678100870b; no voters:
I20240507 02:11:03.598282 17460 leader_election.cc:288] T 00000000000000000000000000000000 P aaea554e49f94f188e3a94678100870b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240507 02:11:03.598500 17460 sys_catalog.cc:564] T 00000000000000000000000000000000 P aaea554e49f94f188e3a94678100870b [sys.catalog]: configured and running, proceeding with master startup.
I20240507 02:11:03.599217 17463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:03.595455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43436 (local address 127.31.131.126:33953)
0507 02:11:03.595625 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:03.595632 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:03.595651 (+ 19us) server_negotiation.cc:407] Connection header received
0507 02:11:03.596139 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:03.596145 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:03.596198 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:03.596325 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:03.596667 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:03.597366 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:03.597997 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:03.598218 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:03.598643 (+ 425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:03.598650 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:03.598651 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0507 02:11:03.598778 (+ 127us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:03.598802 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0507 02:11:03.598808 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:03.598861 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:03.598939 (+ 78us) server_negotiation.cc:299] Negotiation successful
0507 02:11:03.598983 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":61,"threads_started":1}
I20240507 02:11:03.598516 17466 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aaea554e49f94f188e3a94678100870b [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:11:03.599712 17466 raft_consensus.cc:686] T 00000000000000000000000000000000 P aaea554e49f94f188e3a94678100870b [term 1 LEADER]: Becoming Leader. State: Replica: aaea554e49f94f188e3a94678100870b, State: Running, Role: LEADER
I20240507 02:11:03.599826 17466 consensus_queue.cc:237] T 00000000000000000000000000000000 P aaea554e49f94f188e3a94678100870b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaea554e49f94f188e3a94678100870b" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 33953 } }
I20240507 02:11:03.600168 17477 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaea554e49f94f188e3a94678100870b [sys.catalog]: SysCatalogTable state changed. Reason: New leader aaea554e49f94f188e3a94678100870b. Latest consensus state: current_term: 1 leader_uuid: "aaea554e49f94f188e3a94678100870b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaea554e49f94f188e3a94678100870b" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 33953 } } }
I20240507 02:11:03.600247 17465 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaea554e49f94f188e3a94678100870b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aaea554e49f94f188e3a94678100870b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaea554e49f94f188e3a94678100870b" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 33953 } } }
I20240507 02:11:03.600479 17465 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaea554e49f94f188e3a94678100870b [sys.catalog]: This master's current role is: LEADER
I20240507 02:11:03.600625 17477 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaea554e49f94f188e3a94678100870b [sys.catalog]: This master's current role is: LEADER
W20240507 02:11:03.600703 17479 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aaea554e49f94f188e3a94678100870b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240507 02:11:03.600745 17479 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240507 02:11:03.600786 17482 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 02:11:03.600921 17482 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240507 02:11:03.602077 17482 catalog_manager.cc:1347] Generated new cluster ID: 727f49f07b954b9e9b2bb141f9e8ea37
I20240507 02:11:03.602120 17482 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240507 02:11:03.610958 17482 catalog_manager.cc:1370] Generated new certificate authority record
I20240507 02:11:03.611408 17482 catalog_manager.cc:1504] Loading token signing keys...
I20240507 02:11:03.619019 17482 catalog_manager.cc:5857] T 00000000000000000000000000000000 P aaea554e49f94f188e3a94678100870b: Generated new TSK 0
I20240507 02:11:03.619163 17482 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240507 02:11:03.623944 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715047604443657-32269-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715047604443657-32269-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715047604443657-32269-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.65:0
--local_ip_for_outbound_sockets=127.31.131.65
--webserver_interface=127.31.131.65
--webserver_port=0
--tserver_master_addrs=127.31.131.126:33953
--builtin_ntp_servers=127.31.131.84:39431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20240507 02:11:03.696959 17484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:11:03.697115 17484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:11:03.697134 17484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:11:03.698366 17484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:11:03.698402 17484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:11:03.698423 17484 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.65
I20240507 02:11:03.699896 17484 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:39431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715047604443657-32269-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715047604443657-32269-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.131.65
--webserver_port=0
--tserver_master_addrs=127.31.131.126:33953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17484
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.65
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715047604443657-32269-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:11:03.700055 17484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:11:03.700258 17484 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:11:03.704321 17490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:11:03.704322 17492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:11:03.704349 17489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:03.704566 17484 server_base.cc:1034] running on GCE node
I20240507 02:11:03.704707 17484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:11:03.704875 17484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:11:03.706004 17484 hybrid_clock.cc:648] HybridClock initialized: now 1715047863705993 us; error 30 us; skew 500 ppm
I20240507 02:11:03.707054 17484 webserver.cc:460] Webserver started at http://127.31.131.65:35703/ using document root <none> and password file <none>
I20240507 02:11:03.707226 17484 fs_manager.cc:359] Metadata directory not provided
I20240507 02:11:03.707262 17484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:11:03.707345 17484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:11:03.708191 17484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715047604443657-32269-0/minicluster-data/ts-0/data/instance:
uuid: "ecd0c3f26e4341d5840e87ab32df8b41"
format_stamp: "Formatted at 2024-05-07 02:11:03 on dist-test-slave-v0bg"
server_key: "ca28151ac1ccf98841497fa34d727fc7"
server_key_iv: "6dc9444a863039e3afdcff571e7157db"
server_key_version: "encryptionkey@0"
I20240507 02:11:03.708436 17484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715047604443657-32269-0/minicluster-data/ts-0/wal/instance:
uuid: "ecd0c3f26e4341d5840e87ab32df8b41"
format_stamp: "Formatted at 2024-05-07 02:11:03 on dist-test-slave-v0bg"
server_key: "ca28151ac1ccf98841497fa34d727fc7"
server_key_iv: "6dc9444a863039e3afdcff571e7157db"
server_key_version: "encryptionkey@0"
I20240507 02:11:03.709434 17484 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240507 02:11:03.710029 17498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:03.710180 17484 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:03.710245 17484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715047604443657-32269-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715047604443657-32269-0/minicluster-data/ts-0/wal
uuid: "ecd0c3f26e4341d5840e87ab32df8b41"
format_stamp: "Formatted at 2024-05-07 02:11:03 on dist-test-slave-v0bg"
server_key: "ca28151ac1ccf98841497fa34d727fc7"
server_key_iv: "6dc9444a863039e3afdcff571e7157db"
server_key_version: "encryptionkey@0"
I20240507 02:11:03.710300 17484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715047604443657-32269-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715047604443657-32269-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715047604443657-32269-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:11:03.724866 17484 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:11:03.725064 17484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:11:03.725153 17484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:11:03.725306 17484 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:11:03.725549 17484 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:11:03.725579 17484 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:03.725605 17484 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:11:03.725634 17484 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:03.730391 17484 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.65:44747
I20240507 02:11:03.730440 17611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.65:44747 every 8 connection(s)
I20240507 02:11:03.730686 17484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
I20240507 02:11:03.734570 17463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:03.731231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:53675 (local address 127.31.131.126:33953)
0507 02:11:03.731268 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:03.731270 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:03.731470 (+ 200us) server_negotiation.cc:407] Connection header received
0507 02:11:03.731883 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:03.731884 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:03.731896 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:03.731944 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:03.732337 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:03.732808 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:03.733299 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:03.733468 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:03.734305 (+ 837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:03.734309 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:03.734309 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0507 02:11:03.734313 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:03.734327 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0507 02:11:03.734330 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:03.734370 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:03.734446 (+ 76us) server_negotiation.cc:299] Negotiation successful
0507 02:11:03.734474 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20240507 02:11:03.734686 17614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:03.731232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:33953 (local address 127.31.131.65:53675)
0507 02:11:03.731381 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:03.731401 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0507 02:11:03.731794 (+ 393us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:11:03.731990 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:11:03.731994 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:11:03.732001 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:11:03.732257 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:03.732259 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:03.732881 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:11:03.732882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:11:03.733223 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:03.733225 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:03.733332 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:03.734185 (+ 853us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0507 02:11:03.734186 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:11:03.734201 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0507 02:11:03.734215 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:11:03.734377 (+ 162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:11:03.734381 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:11:03.734383 (+ 2us) client_negotiation.cc:769] Sending connection context
0507 02:11:03.734442 (+ 59us) client_negotiation.cc:240] Negotiation successful
0507 02:11:03.734501 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":76,"threads_started":1}
I20240507 02:11:03.735122 17612 heartbeater.cc:347] Connected to a master server at 127.31.131.126:33953
I20240507 02:11:03.735213 17612 heartbeater.cc:464] Registering TS with master...
I20240507 02:11:03.735390 17612 heartbeater.cc:511] Master 127.31.131.126:33953 requested a full tablet report, sending...
I20240507 02:11:03.735750 17424 ts_manager.cc:194] Registered new tserver with Master: ecd0c3f26e4341d5840e87ab32df8b41 (127.31.131.65:44747)
I20240507 02:11:03.736333 17424 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.131.65:53675
I20240507 02:11:03.737565 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 17484
I20240507 02:11:03.737651 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715047604443657-32269-0/minicluster-data/ts-0/wal/instance
I20240507 02:11:03.737820 32269 external_mini_cluster.cc:1428] Setting key e0023f30ebe6d3a26b635589675855ed
I20240507 02:11:03.738703 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715047604443657-32269-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715047604443657-32269-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715047604443657-32269-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.66:0
--local_ip_for_outbound_sockets=127.31.131.66
--webserver_interface=127.31.131.66
--webserver_port=0
--tserver_master_addrs=127.31.131.126:33953
--builtin_ntp_servers=127.31.131.84:39431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20240507 02:11:03.810972 17615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:11:03.811115 17615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:11:03.811133 17615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:11:03.812312 17615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:11:03.812345 17615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:11:03.812366 17615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.66
I20240507 02:11:03.813740 17615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:39431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715047604443657-32269-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715047604443657-32269-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.131.66
--webserver_port=0
--tserver_master_addrs=127.31.131.126:33953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.66
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715047604443657-32269-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:11:03.813887 17615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:11:03.814062 17615 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:11:03.817571 17621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:03.817719 17615 server_base.cc:1034] running on GCE node
W20240507 02:11:03.817719 17620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:11:03.817732 17623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:03.817968 17615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:11:03.818156 17615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:11:03.819272 17615 hybrid_clock.cc:648] HybridClock initialized: now 1715047863819257 us; error 26 us; skew 500 ppm
I20240507 02:11:03.820277 17615 webserver.cc:460] Webserver started at http://127.31.131.66:45665/ using document root <none> and password file <none>
I20240507 02:11:03.820430 17615 fs_manager.cc:359] Metadata directory not provided
I20240507 02:11:03.820468 17615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:11:03.820565 17615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:11:03.821398 17615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715047604443657-32269-0/minicluster-data/ts-1/data/instance:
uuid: "5c7dcce37dd7487783efce6c33c3484f"
format_stamp: "Formatted at 2024-05-07 02:11:03 on dist-test-slave-v0bg"
server_key: "9b8aa266df7f3857d9ddfe1a0b5f6b77"
server_key_iv: "9ed74017a0cb96ebabf9df5c4237578b"
server_key_version: "encryptionkey@0"
I20240507 02:11:03.821655 17615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715047604443657-32269-0/minicluster-data/ts-1/wal/instance:
uuid: "5c7dcce37dd7487783efce6c33c3484f"
format_stamp: "Formatted at 2024-05-07 02:11:03 on dist-test-slave-v0bg"
server_key: "9b8aa266df7f3857d9ddfe1a0b5f6b77"
server_key_iv: "9ed74017a0cb96ebabf9df5c4237578b"
server_key_version: "encryptionkey@0"
I20240507 02:11:03.822651 17615 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20240507 02:11:03.823181 17629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:03.823318 17615 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:03.823381 17615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715047604443657-32269-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715047604443657-32269-0/minicluster-data/ts-1/wal
uuid: "5c7dcce37dd7487783efce6c33c3484f"
format_stamp: "Formatted at 2024-05-07 02:11:03 on dist-test-slave-v0bg"
server_key: "9b8aa266df7f3857d9ddfe1a0b5f6b77"
server_key_iv: "9ed74017a0cb96ebabf9df5c4237578b"
server_key_version: "encryptionkey@0"
I20240507 02:11:03.823432 17615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715047604443657-32269-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715047604443657-32269-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715047604443657-32269-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:11:03.833653 17615 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:11:03.833817 17615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:11:03.833901 17615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:11:03.834039 17615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:11:03.834280 17615 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:11:03.834308 17615 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:03.834332 17615 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:11:03.834347 17615 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:03.838961 17615 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.66:37335
I20240507 02:11:03.839002 17742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.66:37335 every 8 connection(s)
I20240507 02:11:03.839249 17615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
I20240507 02:11:03.842150 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 17615
I20240507 02:11:03.842253 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715047604443657-32269-0/minicluster-data/ts-1/wal/instance
I20240507 02:11:03.842432 32269 external_mini_cluster.cc:1428] Setting key b1a0884cf555127df3f7d4302175415d
I20240507 02:11:03.842989 17463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:03.839748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.66:37287 (local address 127.31.131.126:33953)
0507 02:11:03.839791 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:03.839792 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:03.839953 (+ 161us) server_negotiation.cc:407] Connection header received
0507 02:11:03.840347 (+ 394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:03.840349 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:03.840359 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:03.840403 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:03.840805 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:03.841265 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:03.841757 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:03.841913 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:03.842746 (+ 833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:03.842751 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:03.842751 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0507 02:11:03.842755 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:03.842768 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0507 02:11:03.842771 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:03.842813 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:03.842887 (+ 74us) server_negotiation.cc:299] Negotiation successful
0507 02:11:03.842916 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20240507 02:11:03.843125 17745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:03.839732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:33953 (local address 127.31.131.66:37287)
0507 02:11:03.839872 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:03.839890 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0507 02:11:03.840259 (+ 369us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:11:03.840459 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:11:03.840462 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:11:03.840469 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:11:03.840728 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:03.840730 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:03.841340 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:11:03.841341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:11:03.841686 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:03.841687 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:03.841797 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:03.842638 (+ 841us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0507 02:11:03.842638 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:11:03.842655 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0507 02:11:03.842669 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:11:03.842820 (+ 151us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:11:03.842823 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:11:03.842826 (+ 3us) client_negotiation.cc:769] Sending connection context
0507 02:11:03.842887 (+ 61us) client_negotiation.cc:240] Negotiation successful
0507 02:11:03.842942 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":58,"threads_started":1}
I20240507 02:11:03.843253 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715047604443657-32269-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715047604443657-32269-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715047604443657-32269-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.67:0
--local_ip_for_outbound_sockets=127.31.131.67
--webserver_interface=127.31.131.67
--webserver_port=0
--tserver_master_addrs=127.31.131.126:33953
--builtin_ntp_servers=127.31.131.84:39431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20240507 02:11:03.843511 17743 heartbeater.cc:347] Connected to a master server at 127.31.131.126:33953
I20240507 02:11:03.843587 17743 heartbeater.cc:464] Registering TS with master...
I20240507 02:11:03.843755 17743 heartbeater.cc:511] Master 127.31.131.126:33953 requested a full tablet report, sending...
I20240507 02:11:03.844055 17424 ts_manager.cc:194] Registered new tserver with Master: 5c7dcce37dd7487783efce6c33c3484f (127.31.131.66:37335)
I20240507 02:11:03.844485 17424 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.131.66:37287
W20240507 02:11:03.915923 17746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:11:03.916059 17746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:11:03.916085 17746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:11:03.917330 17746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:11:03.917366 17746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:11:03.917393 17746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.67
I20240507 02:11:03.918745 17746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:39431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715047604443657-32269-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715047604443657-32269-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.131.67
--webserver_port=0
--tserver_master_addrs=127.31.131.126:33953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17746
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.67
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715047604443657-32269-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:11:03.918922 17746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:11:03.919121 17746 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:11:03.922508 17751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:11:03.922600 17752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:11:03.922677 17754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:03.922690 17746 server_base.cc:1034] running on GCE node
I20240507 02:11:03.922837 17746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:11:03.923007 17746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:11:03.924134 17746 hybrid_clock.cc:648] HybridClock initialized: now 1715047863924126 us; error 25 us; skew 500 ppm
I20240507 02:11:03.925150 17746 webserver.cc:460] Webserver started at http://127.31.131.67:41729/ using document root <none> and password file <none>
I20240507 02:11:03.925308 17746 fs_manager.cc:359] Metadata directory not provided
I20240507 02:11:03.925345 17746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:11:03.925436 17746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:11:03.926332 17746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715047604443657-32269-0/minicluster-data/ts-2/data/instance:
uuid: "60dea56b6f834a648dce4a3d9474d719"
format_stamp: "Formatted at 2024-05-07 02:11:03 on dist-test-slave-v0bg"
server_key: "695c1fa10f0783e1ed9e30ef30e7be17"
server_key_iv: "c6d1e3ebb5f4e529c088659fd0286daf"
server_key_version: "encryptionkey@0"
I20240507 02:11:03.926645 17746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715047604443657-32269-0/minicluster-data/ts-2/wal/instance:
uuid: "60dea56b6f834a648dce4a3d9474d719"
format_stamp: "Formatted at 2024-05-07 02:11:03 on dist-test-slave-v0bg"
server_key: "695c1fa10f0783e1ed9e30ef30e7be17"
server_key_iv: "c6d1e3ebb5f4e529c088659fd0286daf"
server_key_version: "encryptionkey@0"
I20240507 02:11:03.927613 17746 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240507 02:11:03.928138 17760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:03.928272 17746 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240507 02:11:03.928330 17746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715047604443657-32269-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715047604443657-32269-0/minicluster-data/ts-2/wal
uuid: "60dea56b6f834a648dce4a3d9474d719"
format_stamp: "Formatted at 2024-05-07 02:11:03 on dist-test-slave-v0bg"
server_key: "695c1fa10f0783e1ed9e30ef30e7be17"
server_key_iv: "c6d1e3ebb5f4e529c088659fd0286daf"
server_key_version: "encryptionkey@0"
I20240507 02:11:03.928387 17746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715047604443657-32269-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715047604443657-32269-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715047604443657-32269-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:11:03.946187 17746 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:11:03.946349 17746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:11:03.946434 17746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:11:03.946624 17746 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:11:03.946874 17746 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:11:03.946913 17746 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:03.946939 17746 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:11:03.946964 17746 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:03.951447 17746 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.67:41817
I20240507 02:11:03.951498 17873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.67:41817 every 8 connection(s)
I20240507 02:11:03.951711 17746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
I20240507 02:11:03.955176 17463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:03.952228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.67:58397 (local address 127.31.131.126:33953)
0507 02:11:03.952286 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:03.952287 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:03.952438 (+ 151us) server_negotiation.cc:407] Connection header received
0507 02:11:03.952822 (+ 384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:03.952823 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:03.952832 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:03.952877 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:03.953233 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:03.953683 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:03.954129 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:03.954245 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:03.954967 (+ 722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:03.954970 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:03.954971 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0507 02:11:03.954974 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:03.954987 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0507 02:11:03.954990 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:03.955030 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:03.955090 (+ 60us) server_negotiation.cc:299] Negotiation successful
0507 02:11:03.955117 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20240507 02:11:03.955245 17876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:03.952199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:33953 (local address 127.31.131.67:58397)
0507 02:11:03.952352 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:03.952375 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0507 02:11:03.952748 (+ 373us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:11:03.952919 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:11:03.952922 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:11:03.952929 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:11:03.953181 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:03.953183 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:03.953739 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:11:03.953740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:11:03.954079 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:03.954080 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:03.954160 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:03.954868 (+ 708us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0507 02:11:03.954868 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:11:03.954882 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0507 02:11:03.954896 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:11:03.955037 (+ 141us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:11:03.955039 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:11:03.955040 (+ 1us) client_negotiation.cc:769] Sending connection context
0507 02:11:03.955086 (+ 46us) client_negotiation.cc:240] Negotiation successful
0507 02:11:03.955130 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":70,"threads_started":1}
I20240507 02:11:03.955610 17874 heartbeater.cc:347] Connected to a master server at 127.31.131.126:33953
I20240507 02:11:03.955680 17874 heartbeater.cc:464] Registering TS with master...
I20240507 02:11:03.955840 17874 heartbeater.cc:511] Master 127.31.131.126:33953 requested a full tablet report, sending...
I20240507 02:11:03.956133 17424 ts_manager.cc:194] Registered new tserver with Master: 60dea56b6f834a648dce4a3d9474d719 (127.31.131.67:41817)
I20240507 02:11:03.956472 17424 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.131.67:58397
I20240507 02:11:03.957125 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 17746
I20240507 02:11:03.957207 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715047604443657-32269-0/minicluster-data/ts-2/wal/instance
I20240507 02:11:03.957363 32269 external_mini_cluster.cc:1428] Setting key 4376358b252da9cbc7b41ac51acd943d
I20240507 02:11:03.958058 32269 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240507 02:11:03.961282 17463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:03.959211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43452 (local address 127.31.131.126:33953)
0507 02:11:03.959260 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:03.959262 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:03.959328 (+ 66us) server_negotiation.cc:407] Connection header received
0507 02:11:03.959387 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:03.959388 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:03.959395 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:03.959445 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:03.959714 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:03.960139 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:03.960638 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:03.960787 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:03.961080 (+ 293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:03.961083 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:03.961084 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0507 02:11:03.961087 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:03.961097 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0507 02:11:03.961100 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:03.961140 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:03.961198 (+ 58us) server_negotiation.cc:299] Negotiation successful
0507 02:11:03.961217 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20240507 02:11:03.962469 17424 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43452:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 02:11:03.963788 17424 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 02:11:03.968273 17885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:03.965397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.65:44747 (local address 127.0.0.1:57712)
0507 02:11:03.965640 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:03.965648 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0507 02:11:03.965695 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:11:03.965953 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:11:03.965954 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:11:03.965955 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:11:03.966128 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:03.966129 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:03.967215 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:11:03.967216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:11:03.967922 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:03.967923 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:03.967987 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:03.968013 (+ 26us) client_negotiation.cc:769] Sending connection context
0507 02:11:03.968059 (+ 46us) client_negotiation.cc:240] Negotiation successful
0507 02:11:03.968209 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":85,"threads_started":1}
I20240507 02:11:03.968497 17883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:03.965347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.66:37335 (local address 127.0.0.1:60852)
0507 02:11:03.965570 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:03.965587 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0507 02:11:03.965650 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:11:03.965879 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:11:03.965882 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:11:03.965884 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:11:03.966119 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:03.966121 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:03.967530 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:11:03.967531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:11:03.968322 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:03.968325 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:03.968382 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:03.968385 (+ 3us) client_negotiation.cc:769] Sending connection context
0507 02:11:03.968406 (+ 21us) client_negotiation.cc:240] Negotiation successful
0507 02:11:03.968435 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":41,"threads_started":1}
I20240507 02:11:03.968652 17886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:03.965508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57712 (local address 127.31.131.65:44747)
0507 02:11:03.965803 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:03.965813 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:03.965822 (+ 9us) server_negotiation.cc:407] Connection header received
0507 02:11:03.965858 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:03.965859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:03.965874 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:03.965950 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:11:03.966196 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:03.967142 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:03.968003 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:03.968435 (+ 432us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:03.968522 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:03.968548 (+ 26us) server_negotiation.cc:299] Negotiation successful
0507 02:11:03.968582 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":85,"threads_started":1}
I20240507 02:11:03.969111 17884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:03.965426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60852 (local address 127.31.131.66:37335)
0507 02:11:03.965690 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:03.965697 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:03.965707 (+ 10us) server_negotiation.cc:407] Connection header received
0507 02:11:03.965745 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:03.965747 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:03.965764 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:03.966195 (+ 431us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:11:03.966267 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:03.967452 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:03.968435 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:03.968908 (+ 473us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:03.968976 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:03.969012 (+ 36us) server_negotiation.cc:299] Negotiation successful
0507 02:11:03.969045 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":80,"threads_started":1}
I20240507 02:11:03.969120 17546 tablet_service.cc:1467] Processing CreateTablet for tablet fd71912d140741f889ba68d6d3168eeb (DEFAULT_TABLE table=test-table [id=9e7403b5c7b9446e843c803a1d593749]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:11:03.969352 17546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd71912d140741f889ba68d6d3168eeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:11:03.969471 17882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:03.965519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.67:41817 (local address 127.0.0.1:39292)
0507 02:11:03.965795 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:03.965804 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0507 02:11:03.965854 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:11:03.966034 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:11:03.966035 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:11:03.966037 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:11:03.966170 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:03.966171 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:03.968623 (+ 2452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:11:03.968624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:11:03.969284 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:03.969287 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:03.969349 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:03.969352 (+ 3us) client_negotiation.cc:769] Sending connection context
0507 02:11:03.969380 (+ 28us) client_negotiation.cc:240] Negotiation successful
0507 02:11:03.969412 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":73,"threads_started":1}
I20240507 02:11:03.969609 17677 tablet_service.cc:1467] Processing CreateTablet for tablet fd71912d140741f889ba68d6d3168eeb (DEFAULT_TABLE table=test-table [id=9e7403b5c7b9446e843c803a1d593749]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:11:03.969846 17677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd71912d140741f889ba68d6d3168eeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:11:03.970155 17887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:03.965674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39292 (local address 127.31.131.67:41817)
0507 02:11:03.965802 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:03.965809 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:03.965844 (+ 35us) server_negotiation.cc:407] Connection header received
0507 02:11:03.965907 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:03.965909 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:03.965921 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:03.965982 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:11:03.966319 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:03.967457 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:03.969601 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:03.970013 (+ 412us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:03.970059 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:03.970084 (+ 25us) server_negotiation.cc:299] Negotiation successful
0507 02:11:03.970106 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":69,"threads_started":1}
I20240507 02:11:03.970535 17808 tablet_service.cc:1467] Processing CreateTablet for tablet fd71912d140741f889ba68d6d3168eeb (DEFAULT_TABLE table=test-table [id=9e7403b5c7b9446e843c803a1d593749]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:11:03.970795 17808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd71912d140741f889ba68d6d3168eeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:11:03.972182 17888 tablet_bootstrap.cc:492] T fd71912d140741f889ba68d6d3168eeb P ecd0c3f26e4341d5840e87ab32df8b41: Bootstrap starting.
I20240507 02:11:03.972577 17889 tablet_bootstrap.cc:492] T fd71912d140741f889ba68d6d3168eeb P 5c7dcce37dd7487783efce6c33c3484f: Bootstrap starting.
I20240507 02:11:03.972805 17888 tablet_bootstrap.cc:654] T fd71912d140741f889ba68d6d3168eeb P ecd0c3f26e4341d5840e87ab32df8b41: Neither blocks nor log segments found. Creating new log.
I20240507 02:11:03.973043 17889 tablet_bootstrap.cc:654] T fd71912d140741f889ba68d6d3168eeb P 5c7dcce37dd7487783efce6c33c3484f: Neither blocks nor log segments found. Creating new log.
I20240507 02:11:03.973035 17890 tablet_bootstrap.cc:492] T fd71912d140741f889ba68d6d3168eeb P 60dea56b6f834a648dce4a3d9474d719: Bootstrap starting.
I20240507 02:11:03.973083 17888 log.cc:824] T fd71912d140741f889ba68d6d3168eeb P ecd0c3f26e4341d5840e87ab32df8b41: Log is configured to *not* fsync() on all Append() calls
I20240507 02:11:03.973301 17889 log.cc:824] T fd71912d140741f889ba68d6d3168eeb P 5c7dcce37dd7487783efce6c33c3484f: Log is configured to *not* fsync() on all Append() calls
I20240507 02:11:03.973587 17890 tablet_bootstrap.cc:654] T fd71912d140741f889ba68d6d3168eeb P 60dea56b6f834a648dce4a3d9474d719: Neither blocks nor log segments found. Creating new log.
I20240507 02:11:03.973822 17890 log.cc:824] T fd71912d140741f889ba68d6d3168eeb P 60dea56b6f834a648dce4a3d9474d719: Log is configured to *not* fsync() on all Append() calls
I20240507 02:11:03.973986 17889 tablet_bootstrap.cc:492] T fd71912d140741f889ba68d6d3168eeb P 5c7dcce37dd7487783efce6c33c3484f: No bootstrap required, opened a new log
I20240507 02:11:03.974051 17889 ts_tablet_manager.cc:1367] T fd71912d140741f889ba68d6d3168eeb P 5c7dcce37dd7487783efce6c33c3484f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240507 02:11:03.974443 17888 tablet_bootstrap.cc:492] T fd71912d140741f889ba68d6d3168eeb P ecd0c3f26e4341d5840e87ab32df8b41: No bootstrap required, opened a new log
I20240507 02:11:03.974450 17890 tablet_bootstrap.cc:492] T fd71912d140741f889ba68d6d3168eeb P 60dea56b6f834a648dce4a3d9474d719: No bootstrap required, opened a new log
I20240507 02:11:03.974507 17888 ts_tablet_manager.cc:1367] T fd71912d140741f889ba68d6d3168eeb P ecd0c3f26e4341d5840e87ab32df8b41: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240507 02:11:03.974507 17890 ts_tablet_manager.cc:1367] T fd71912d140741f889ba68d6d3168eeb P 60dea56b6f834a648dce4a3d9474d719: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240507 02:11:03.975620 17889 raft_consensus.cc:348] T fd71912d140741f889ba68d6d3168eeb P 5c7dcce37dd7487783efce6c33c3484f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c7dcce37dd7487783efce6c33c3484f" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 37335 } } peers { permanent_uuid: "60dea56b6f834a648dce4a3d9474d719" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 41817 } } peers { permanent_uuid: "ecd0c3f26e4341d5840e87ab32df8b41" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 44747 } }
I20240507 02:11:03.975719 17889 raft_consensus.cc:374] T fd71912d140741f889ba68d6d3168eeb P 5c7dcce37dd7487783efce6c33c3484f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:11:03.975741 17889 raft_consensus.cc:729] T fd71912d140741f889ba68d6d3168eeb P 5c7dcce37dd7487783efce6c33c3484f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c7dcce37dd7487783efce6c33c3484f, State: Initialized, Role: FOLLOWER
I20240507 02:11:03.975852 17889 consensus_queue.cc:260] T fd71912d140741f889ba68d6d3168eeb P 5c7dcce37dd7487783efce6c33c3484f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c7dcce37dd7487783efce6c33c3484f" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 37335 } } peers { permanent_uuid: "60dea56b6f834a648dce4a3d9474d719" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 41817 } } peers { permanent_uuid: "ecd0c3f26e4341d5840e87ab32df8b41" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 44747 } }
I20240507 02:11:03.976052 17889 ts_tablet_manager.cc:1398] T fd71912d140741f889ba68d6d3168eeb P 5c7dcce37dd7487783efce6c33c3484f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240507 02:11:03.976199 17743 heartbeater.cc:503] Master 127.31.131.126:33953 was elected leader, sending a full tablet report...
I20240507 02:11:03.976541 17890 raft_consensus.cc:348] T fd71912d140741f889ba68d6d3168eeb P 60dea56b6f834a648dce4a3d9474d719 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c7dcce37dd7487783efce6c33c3484f" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 37335 } } peers { permanent_uuid: "60dea56b6f834a648dce4a3d9474d719" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 41817 } } peers { permanent_uuid: "ecd0c3f26e4341d5840e87ab32df8b41" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 44747 } }
I20240507 02:11:03.976660 17890 raft_consensus.cc:374] T fd71912d140741f889ba68d6d3168eeb P 60dea56b6f834a648dce4a3d9474d719 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:11:03.976693 17890 raft_consensus.cc:729] T fd71912d140741f889ba68d6d3168eeb P 60dea56b6f834a648dce4a3d9474d719 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60dea56b6f834a648dce4a3d9474d719, State: Initialized, Role: FOLLOWER
I20240507 02:11:03.976697 17888 raft_consensus.cc:348] T fd71912d140741f889ba68d6d3168eeb P ecd0c3f26e4341d5840e87ab32df8b41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c7dcce37dd7487783efce6c33c3484f" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 37335 } } peers { permanent_uuid: "60dea56b6f834a648dce4a3d9474d719" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 41817 } } peers { permanent_uuid: "ecd0c3f26e4341d5840e87ab32df8b41" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 44747 } }
I20240507 02:11:03.976797 17888 raft_consensus.cc:374] T fd71912d140741f889ba68d6d3168eeb P ecd0c3f26e4341d5840e87ab32df8b41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:11:03.976819 17888 raft_consensus.cc:729] T fd71912d140741f889ba68d6d3168eeb P ecd0c3f26e4341d5840e87ab32df8b41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecd0c3f26e4341d5840e87ab32df8b41, State: Initialized, Role: FOLLOWER
I20240507 02:11:03.976785 17890 consensus_queue.cc:260] T fd71912d140741f889ba68d6d3168eeb P 60dea56b6f834a648dce4a3d9474d719 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c7dcce37dd7487783efce6c33c3484f" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 37335 } } peers { permanent_uuid: "60dea56b6f834a648dce4a3d9474d719" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 41817 } } peers { permanent_uuid: "ecd0c3f26e4341d5840e87ab32df8b41" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 44747 } }
I20240507 02:11:03.976903 17888 consensus_queue.cc:260] T fd71912d140741f889ba68d6d3168eeb P ecd0c3f26e4341d5840e87ab32df8b41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c7dcce37dd7487783efce6c33c3484f" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 37335 } } peers { permanent_uuid: "60dea56b6f834a648dce4a3d9474d719" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 41817 } } peers { permanent_uuid: "ecd0c3f26e4341d5840e87ab32df8b41" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 44747 } }
I20240507 02:11:03.977003 17890 ts_tablet_manager.cc:1398] T fd71912d140741f889ba68d6d3168eeb P 60dea56b6f834a648dce4a3d9474d719: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240507 02:11:03.977097 17888 ts_tablet_manager.cc:1398] T fd71912d140741f889ba68d6d3168eeb P ecd0c3f26e4341d5840e87ab32df8b41: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240507 02:11:03.977118 17874 heartbeater.cc:503] Master 127.31.131.126:33953 was elected leader, sending a full tablet report...
I20240507 02:11:03.977198 17612 heartbeater.cc:503] Master 127.31.131.126:33953 was elected leader, sending a full tablet report...
W20240507 02:11:03.984047 17613 tablet.cc:2369] T fd71912d140741f889ba68d6d3168eeb P ecd0c3f26e4341d5840e87ab32df8b41: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240507 02:11:04.056236 17894 raft_consensus.cc:482] T fd71912d140741f889ba68d6d3168eeb P 5c7dcce37dd7487783efce6c33c3484f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 02:11:04.056324 17894 raft_consensus.cc:504] T fd71912d140741f889ba68d6d3168eeb P 5c7dcce37dd7487783efce6c33c3484f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c7dcce37dd7487783efce6c33c3484f" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 37335 } } peers { permanent_uuid: "60dea56b6f834a648dce4a3d9474d719" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 41817 } } peers { permanent_uuid: "ecd0c3f26e4341d5840e87ab32df8b41" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 44747 } }
I20240507 02:11:04.056615 17894 leader_election.cc:288] T fd71912d140741f889ba68d6d3168eeb P 5c7dcce37dd7487783efce6c33c3484f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60dea56b6f834a648dce4a3d9474d719 (127.31.131.67:41817), ecd0c3f26e4341d5840e87ab32df8b41 (127.31.131.65:44747)
I20240507 02:11:04.058846 17745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:04.056742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.67:41817 (local address 127.31.131.66:43349)
0507 02:11:04.056815 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:04.056827 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0507 02:11:04.056866 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:11:04.056999 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:11:04.057000 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:11:04.057002 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:11:04.057274 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:04.057277 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:04.057923 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:11:04.057924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:11:04.058630 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:04.058632 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:04.058697 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:04.058700 (+ 3us) client_negotiation.cc:769] Sending connection context
0507 02:11:04.058729 (+ 29us) client_negotiation.cc:240] Negotiation successful
0507 02:11:04.058767 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20240507 02:11:04.058853 17897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:04.056748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.65:44747 (local address 127.31.131.66:44379)
0507 02:11:04.056929 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:04.056938 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0507 02:11:04.056996 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:11:04.057126 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:11:04.057126 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:11:04.057128 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:11:04.057302 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:04.057303 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:04.057934 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:11:04.057935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:11:04.058598 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:04.058600 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:04.058667 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:04.058671 (+ 4us) client_negotiation.cc:769] Sending connection context
0507 02:11:04.058709 (+ 38us) client_negotiation.cc:240] Negotiation successful
0507 02:11:04.058744 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":16,"thread_start_us":92,"threads_started":1}
I20240507 02:11:04.059247 17886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:04.056831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.66:44379 (local address 127.31.131.65:44747)
0507 02:11:04.056871 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:04.056873 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:04.057013 (+ 140us) server_negotiation.cc:407] Connection header received
0507 02:11:04.057040 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:04.057041 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:04.057054 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:04.057121 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:11:04.057387 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:04.057861 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:04.058671 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:04.059107 (+ 436us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:04.059137 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:04.059159 (+ 22us) server_negotiation.cc:299] Negotiation successful
0507 02:11:04.059192 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20240507 02:11:04.059285 17887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:04.056809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.66:43349 (local address 127.31.131.67:41817)
0507 02:11:04.056851 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:04.056853 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:04.056862 (+ 9us) server_negotiation.cc:407] Connection header received
0507 02:11:04.056920 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:04.056921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:04.056932 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:04.056993 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:11:04.057371 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:04.057839 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:04.058718 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:04.059194 (+ 476us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:04.059202 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:04.059221 (+ 19us) server_negotiation.cc:299] Negotiation successful
0507 02:11:04.059242 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20240507 02:11:04.059489 17566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd71912d140741f889ba68d6d3168eeb" candidate_uuid: "5c7dcce37dd7487783efce6c33c3484f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecd0c3f26e4341d5840e87ab32df8b41" is_pre_election: true
I20240507 02:11:04.059540 17828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd71912d140741f889ba68d6d3168eeb" candidate_uuid: "5c7dcce37dd7487783efce6c33c3484f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60dea56b6f834a648dce4a3d9474d719" is_pre_election: true
I20240507 02:11:04.059615 17566 raft_consensus.cc:2454] T fd71912d140741f889ba68d6d3168eeb P ecd0c3f26e4341d5840e87ab32df8b41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c7dcce37dd7487783efce6c33c3484f in term 0.
I20240507 02:11:04.059643 17828 raft_consensus.cc:2454] T fd71912d140741f889ba68d6d3168eeb P 60dea56b6f834a648dce4a3d9474d719 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c7dcce37dd7487783efce6c33c3484f in term 0.
I20240507 02:11:04.059819 17631 leader_election.cc:302] T fd71912d140741f889ba68d6d3168eeb P 5c7dcce37dd7487783efce6c33c3484f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c7dcce37dd7487783efce6c33c3484f, 60dea56b6f834a648dce4a3d9474d719; no voters:
I20240507 02:11:04.059957 17894 raft_consensus.cc:2781] T fd71912d140741f889ba68d6d3168eeb P 5c7dcce37dd7487783efce6c33c3484f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 02:11:04.060026 17894 raft_consensus.cc:482] T fd71912d140741f889ba68d6d3168eeb P 5c7dcce37dd7487783efce6c33c3484f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 02:11:04.060048 17894 raft_consensus.cc:3037] T fd71912d140741f889ba68d6d3168eeb P 5c7dcce37dd7487783efce6c33c3484f [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:11:04.060634 17894 raft_consensus.cc:504] T fd71912d140741f889ba68d6d3168eeb P 5c7dcce37dd7487783efce6c33c3484f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c7dcce37dd7487783efce6c33c3484f" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 37335 } } peers { permanent_uuid: "60dea56b6f834a648dce4a3d9474d719" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 41817 } } peers { permanent_uuid: "ecd0c3f26e4341d5840e87ab32df8b41" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 44747 } }
I20240507 02:11:04.060751 17894 leader_election.cc:288] T fd71912d140741f889ba68d6d3168eeb P 5c7dcce37dd7487783efce6c33c3484f [CANDIDATE]: Term 1 election: Requested vote from peers 60dea56b6f834a648dce4a3d9474d719 (127.31.131.67:41817), ecd0c3f26e4341d5840e87ab32df8b41 (127.31.131.65:44747)
I20240507 02:11:04.060894 17828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd71912d140741f889ba68d6d3168eeb" candidate_uuid: "5c7dcce37dd7487783efce6c33c3484f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60dea56b6f834a648dce4a3d9474d719"
I20240507 02:11:04.060953 17828 raft_consensus.cc:3037] T fd71912d140741f889ba68d6d3168eeb P 60dea56b6f834a648dce4a3d9474d719 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:11:04.061535 17828 raft_consensus.cc:2454] T fd71912d140741f889ba68d6d3168eeb P 60dea56b6f834a648dce4a3d9474d719 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c7dcce37dd7487783efce6c33c3484f in term 1.
I20240507 02:11:04.061628 17566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd71912d140741f889ba68d6d3168eeb" candidate_uuid: "5c7dcce37dd7487783efce6c33c3484f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecd0c3f26e4341d5840e87ab32df8b41"
I20240507 02:11:04.061690 17631 leader_election.cc:302] T fd71912d140741f889ba68d6d3168eeb P 5c7dcce37dd7487783efce6c33c3484f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c7dcce37dd7487783efce6c33c3484f, 60dea56b6f834a648dce4a3d9474d719; no voters:
I20240507 02:11:04.061702 17566 raft_consensus.cc:3037] T fd71912d140741f889ba68d6d3168eeb P ecd0c3f26e4341d5840e87ab32df8b41 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:11:04.061762 17894 raft_consensus.cc:2781] T fd71912d140741f889ba68d6d3168eeb P 5c7dcce37dd7487783efce6c33c3484f [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:11:04.061878 17894 raft_consensus.cc:686] T fd71912d140741f889ba68d6d3168eeb P 5c7dcce37dd7487783efce6c33c3484f [term 1 LEADER]: Becoming Leader. State: Replica: 5c7dcce37dd7487783efce6c33c3484f, State: Running, Role: LEADER
I20240507 02:11:04.061969 17894 consensus_queue.cc:237] T fd71912d140741f889ba68d6d3168eeb P 5c7dcce37dd7487783efce6c33c3484f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c7dcce37dd7487783efce6c33c3484f" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 37335 } } peers { permanent_uuid: "60dea56b6f834a648dce4a3d9474d719" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 41817 } } peers { permanent_uuid: "ecd0c3f26e4341d5840e87ab32df8b41" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 44747 } }
I20240507 02:11:04.062424 17566 raft_consensus.cc:2454] T fd71912d140741f889ba68d6d3168eeb P ecd0c3f26e4341d5840e87ab32df8b41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c7dcce37dd7487783efce6c33c3484f in term 1.
I20240507 02:11:04.062665 17424 catalog_manager.cc:5484] T fd71912d140741f889ba68d6d3168eeb P 5c7dcce37dd7487783efce6c33c3484f reported cstate change: term changed from 0 to 1, leader changed from <none> to 5c7dcce37dd7487783efce6c33c3484f (127.31.131.66). New cstate: current_term: 1 leader_uuid: "5c7dcce37dd7487783efce6c33c3484f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c7dcce37dd7487783efce6c33c3484f" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 37335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "60dea56b6f834a648dce4a3d9474d719" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 41817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecd0c3f26e4341d5840e87ab32df8b41" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 44747 } health_report { overall_health: UNKNOWN } } }
I20240507 02:11:04.081477 17884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:04.079781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60860 (local address 127.31.131.66:37335)
0507 02:11:04.079826 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:04.079828 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:04.079834 (+ 6us) server_negotiation.cc:407] Connection header received
0507 02:11:04.079858 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:04.079858 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:04.079872 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:04.079931 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0507 02:11:04.080127 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:04.080556 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:04.081056 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:04.081187 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:04.081218 (+ 31us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0507 02:11:04.081289 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0507 02:11:04.081352 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:04.081403 (+ 51us) server_negotiation.cc:299] Negotiation successful
0507 02:11:04.081423 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20240507 02:11:04.082859 17566 raft_consensus.cc:1261] T fd71912d140741f889ba68d6d3168eeb P ecd0c3f26e4341d5840e87ab32df8b41 [term 1 FOLLOWER]: Refusing update from remote peer 5c7dcce37dd7487783efce6c33c3484f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 02:11:04.082851 17828 raft_consensus.cc:1261] T fd71912d140741f889ba68d6d3168eeb P 60dea56b6f834a648dce4a3d9474d719 [term 1 FOLLOWER]: Refusing update from remote peer 5c7dcce37dd7487783efce6c33c3484f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 02:11:04.083127 17894 consensus_queue.cc:1035] T fd71912d140741f889ba68d6d3168eeb P 5c7dcce37dd7487783efce6c33c3484f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60dea56b6f834a648dce4a3d9474d719" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 41817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 02:11:04.083249 17898 consensus_queue.cc:1035] T fd71912d140741f889ba68d6d3168eeb P 5c7dcce37dd7487783efce6c33c3484f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecd0c3f26e4341d5840e87ab32df8b41" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 44747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 02:11:04.086745 17902 mvcc.cc:205] Tried to move back new op lower bound from 7024836051281313792 to 7024836051198328832. Current Snapshot: MvccSnapshot[applied={T|T < 7024836051281313792}]
I20240507 02:11:04.086747 17424 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43452:
table { table_name: "test-table" } modify_external_catalogs: true
I20240507 02:11:04.086819 17424 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43452:
table { table_name: "test-table" } modify_external_catalogs: true
I20240507 02:11:04.087474 17424 catalog_manager.cc:5771] T 00000000000000000000000000000000 P aaea554e49f94f188e3a94678100870b: Sending DeleteTablet for 3 replicas of tablet fd71912d140741f889ba68d6d3168eeb
I20240507 02:11:04.087746 17808 tablet_service.cc:1514] Processing DeleteTablet for tablet fd71912d140741f889ba68d6d3168eeb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:11:04 UTC) from {username='slave'} at 127.0.0.1:39292
I20240507 02:11:04.087777 17677 tablet_service.cc:1514] Processing DeleteTablet for tablet fd71912d140741f889ba68d6d3168eeb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:11:04 UTC) from {username='slave'} at 127.0.0.1:60852
I20240507 02:11:04.087764 17807 tablet_service.cc:1514] Processing DeleteTablet for tablet fd71912d140741f889ba68d6d3168eeb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:11:04 UTC) from {username='slave'} at 127.0.0.1:39292
I20240507 02:11:04.087961 17911 tablet_replica.cc:330] T fd71912d140741f889ba68d6d3168eeb P 5c7dcce37dd7487783efce6c33c3484f: stopping tablet replica
I20240507 02:11:04.087961 17546 tablet_service.cc:1514] Processing DeleteTablet for tablet fd71912d140741f889ba68d6d3168eeb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:11:04 UTC) from {username='slave'} at 127.0.0.1:57712
I20240507 02:11:04.088040 17911 raft_consensus.cc:2229] T fd71912d140741f889ba68d6d3168eeb P 5c7dcce37dd7487783efce6c33c3484f [term 1 LEADER]: Raft consensus shutting down.
I20240507 02:11:04.088116 17911 raft_consensus.cc:2258] T fd71912d140741f889ba68d6d3168eeb P 5c7dcce37dd7487783efce6c33c3484f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 02:11:04.088136 17912 tablet_replica.cc:330] T fd71912d140741f889ba68d6d3168eeb P 60dea56b6f834a648dce4a3d9474d719: stopping tablet replica
I20240507 02:11:04.088205 17912 raft_consensus.cc:2229] T fd71912d140741f889ba68d6d3168eeb P 60dea56b6f834a648dce4a3d9474d719 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 02:11:04.088258 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 17484
I20240507 02:11:04.088289 17912 raft_consensus.cc:2258] T fd71912d140741f889ba68d6d3168eeb P 60dea56b6f834a648dce4a3d9474d719 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 02:11:04.089205 17911 ts_tablet_manager.cc:1875] T fd71912d140741f889ba68d6d3168eeb P 5c7dcce37dd7487783efce6c33c3484f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 02:11:04.090860 17911 ts_tablet_manager.cc:1888] T fd71912d140741f889ba68d6d3168eeb P 5c7dcce37dd7487783efce6c33c3484f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240507 02:11:04.090966 17911 log.cc:1196] T fd71912d140741f889ba68d6d3168eeb P 5c7dcce37dd7487783efce6c33c3484f: Deleting WAL directory at /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715047604443657-32269-0/minicluster-data/ts-1/wal/wals/fd71912d140741f889ba68d6d3168eeb
I20240507 02:11:04.091274 17911 ts_tablet_manager.cc:1909] T fd71912d140741f889ba68d6d3168eeb P 5c7dcce37dd7487783efce6c33c3484f: Deleting consensus metadata
I20240507 02:11:04.091665 17409 catalog_manager.cc:4830] TS 5c7dcce37dd7487783efce6c33c3484f (127.31.131.66:37335): tablet fd71912d140741f889ba68d6d3168eeb (table test-table [id=9e7403b5c7b9446e843c803a1d593749]) successfully deleted
W20240507 02:11:04.093497 17410 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.131.65:44747 (error 108)
I20240507 02:11:04.093851 17912 ts_tablet_manager.cc:1875] T fd71912d140741f889ba68d6d3168eeb P 60dea56b6f834a648dce4a3d9474d719: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 02:11:04.094902 17912 ts_tablet_manager.cc:1888] T fd71912d140741f889ba68d6d3168eeb P 60dea56b6f834a648dce4a3d9474d719: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240507 02:11:04.094952 17912 log.cc:1196] T fd71912d140741f889ba68d6d3168eeb P 60dea56b6f834a648dce4a3d9474d719: Deleting WAL directory at /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1715047604443657-32269-0/minicluster-data/ts-2/wal/wals/fd71912d140741f889ba68d6d3168eeb
I20240507 02:11:04.095146 17912 ts_tablet_manager.cc:1909] T fd71912d140741f889ba68d6d3168eeb P 60dea56b6f834a648dce4a3d9474d719: Deleting consensus metadata
I20240507 02:11:04.095440 17409 catalog_manager.cc:4830] TS 60dea56b6f834a648dce4a3d9474d719 (127.31.131.67:41817): tablet fd71912d140741f889ba68d6d3168eeb (table test-table [id=9e7403b5c7b9446e843c803a1d593749]) successfully deleted
W20240507 02:11:04.095525 17882 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:11:04.093792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.65:44747 (local address 127.0.0.1:57720)
0507 02:11:04.093839 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:04.093849 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0507 02:11:04.093891 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:11:04.095484 (+ 1593us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.65:44747: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":33}
W20240507 02:11:04.095556 17409 catalog_manager.cc:4797] TS 60dea56b6f834a648dce4a3d9474d719 (127.31.131.67:41817): delete failed for tablet fd71912d140741f889ba68d6d3168eeb because the tablet was not found. No further retry: Not found: Tablet not found: fd71912d140741f889ba68d6d3168eeb
W20240507 02:11:04.095676 17410 catalog_manager.cc:4573] TS ecd0c3f26e4341d5840e87ab32df8b41 (127.31.131.65:44747): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fd71912d140741f889ba68d6d3168eeb: Network error: Client connection negotiation failed: client connection to 127.31.131.65:44747: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240507 02:11:04.095885 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 17615
I20240507 02:11:04.100674 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 17746
I20240507 02:11:04.105154 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 17393
2024-05-07T02:11:04Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (636 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 07 02:11:04 dist-test-slave-v0bg krb5kdc[17917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 07 02:11:04 dist-test-slave-v0bg krb5kdc[17917](info): set up 2 sockets
May 07 02:11:04 dist-test-slave-v0bg krb5kdc[17917](info): commencing operation
krb5kdc: starting...
W20240507 02:11:04.978943 32269 mini_kdc.cc:121] Time spent starting KDC: real 0.857s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/krb5kdc/test-user.keytab.
May 07 02:11:05 dist-test-slave-v0bg krb5kdc[17917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-07T02:11:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-07T02:11:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.131.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab.
Entry for principal kudu/127.31.131.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab.
I20240507 02:11:05.075059 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:39245
--webserver_interface=127.31.131.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab
--principal=kudu/127.31.131.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:41763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.131.126:39245
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:11:05.149448 17930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:11:05.149624 17930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:11:05.149646 17930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:11:05.150897 17930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240507 02:11:05.150938 17930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:11:05.150952 17930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:11:05.150964 17930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240507 02:11:05.150991 17930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240507 02:11:05.152520 17930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:41763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.131.126:39245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab
--principal=kudu/127.31.131.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:39245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.131.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:11:05.152702 17930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:11:05.152871 17930 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:11:05.156287 17936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:05.156459 17930 server_base.cc:1034] running on GCE node
W20240507 02:11:05.156394 17935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:11:05.156456 17938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:05.156797 17930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:11:05.156975 17930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:11:05.158102 17930 hybrid_clock.cc:648] HybridClock initialized: now 1715047865158088 us; error 30 us; skew 500 ppm
May 07 02:11:05 dist-test-slave-v0bg krb5kdc[17917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047865, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:11:05.165793 17930 init.cc:377] Logged in from keytab as kudu/127.31.131.126@KRBTEST.COM (short username kudu)
I20240507 02:11:05.166146 17930 webserver.cc:460] Webserver started at http://127.31.131.126:32917/ using document root <none> and password file <none>
I20240507 02:11:05.166297 17930 fs_manager.cc:359] Metadata directory not provided
I20240507 02:11:05.166334 17930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:11:05.166426 17930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:11:05.167304 17930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/minicluster-data/master-0/data/instance:
uuid: "24ee147d8d914b89882bb72b57c29481"
format_stamp: "Formatted at 2024-05-07 02:11:05 on dist-test-slave-v0bg"
server_key: "1f94cd9ccf56684dcc53965baaa672c9"
server_key_iv: "37907c2ef3036fe3f2656f7281788c9d"
server_key_version: "encryptionkey@0"
I20240507 02:11:05.167572 17930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/minicluster-data/master-0/wal/instance:
uuid: "24ee147d8d914b89882bb72b57c29481"
format_stamp: "Formatted at 2024-05-07 02:11:05 on dist-test-slave-v0bg"
server_key: "1f94cd9ccf56684dcc53965baaa672c9"
server_key_iv: "37907c2ef3036fe3f2656f7281788c9d"
server_key_version: "encryptionkey@0"
I20240507 02:11:05.168586 17930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240507 02:11:05.169126 17945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:05.169258 17930 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240507 02:11:05.169323 17930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/minicluster-data/master-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/minicluster-data/master-0/wal
uuid: "24ee147d8d914b89882bb72b57c29481"
format_stamp: "Formatted at 2024-05-07 02:11:05 on dist-test-slave-v0bg"
server_key: "1f94cd9ccf56684dcc53965baaa672c9"
server_key_iv: "37907c2ef3036fe3f2656f7281788c9d"
server_key_version: "encryptionkey@0"
I20240507 02:11:05.169376 17930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:11:05.188313 17930 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:11:05.190178 17930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:11:05.190286 17930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:11:05.193693 17930 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.126:39245
I20240507 02:11:05.193714 17997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.126:39245 every 8 connection(s)
I20240507 02:11:05.193987 17930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
I20240507 02:11:05.194622 17998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:11:05.196331 17998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24ee147d8d914b89882bb72b57c29481: Bootstrap starting.
I20240507 02:11:05.196776 17998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24ee147d8d914b89882bb72b57c29481: Neither blocks nor log segments found. Creating new log.
I20240507 02:11:05.196978 17998 log.cc:824] T 00000000000000000000000000000000 P 24ee147d8d914b89882bb72b57c29481: Log is configured to *not* fsync() on all Append() calls
I20240507 02:11:05.197492 17998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24ee147d8d914b89882bb72b57c29481: No bootstrap required, opened a new log
I20240507 02:11:05.198359 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 17930
I20240507 02:11:05.198453 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/minicluster-data/master-0/wal/instance
I20240507 02:11:05.198648 32269 external_mini_cluster.cc:1428] Setting key 35bee7b6e57c4267e679bc71808c58e3
I20240507 02:11:05.198890 17998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24ee147d8d914b89882bb72b57c29481 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24ee147d8d914b89882bb72b57c29481" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 39245 } }
I20240507 02:11:05.199057 17998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24ee147d8d914b89882bb72b57c29481 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:11:05.199095 17998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24ee147d8d914b89882bb72b57c29481 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24ee147d8d914b89882bb72b57c29481, State: Initialized, Role: FOLLOWER
I20240507 02:11:05.199185 17998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24ee147d8d914b89882bb72b57c29481 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24ee147d8d914b89882bb72b57c29481" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 39245 } }
I20240507 02:11:05.199256 17998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24ee147d8d914b89882bb72b57c29481 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 02:11:05.199285 17998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24ee147d8d914b89882bb72b57c29481 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 02:11:05.199326 17998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24ee147d8d914b89882bb72b57c29481 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:11:05.199893 17998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24ee147d8d914b89882bb72b57c29481 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24ee147d8d914b89882bb72b57c29481" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 39245 } }
I20240507 02:11:05.199998 17998 leader_election.cc:302] T 00000000000000000000000000000000 P 24ee147d8d914b89882bb72b57c29481 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24ee147d8d914b89882bb72b57c29481; no voters:
I20240507 02:11:05.200136 17998 leader_election.cc:288] T 00000000000000000000000000000000 P 24ee147d8d914b89882bb72b57c29481 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240507 02:11:05.200284 18003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24ee147d8d914b89882bb72b57c29481 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:11:05.200325 17998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24ee147d8d914b89882bb72b57c29481 [sys.catalog]: configured and running, proceeding with master startup.
I20240507 02:11:05.200438 18003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24ee147d8d914b89882bb72b57c29481 [term 1 LEADER]: Becoming Leader. State: Replica: 24ee147d8d914b89882bb72b57c29481, State: Running, Role: LEADER
I20240507 02:11:05.200557 18003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24ee147d8d914b89882bb72b57c29481 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24ee147d8d914b89882bb72b57c29481" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 39245 } }
I20240507 02:11:05.200881 18005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24ee147d8d914b89882bb72b57c29481 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24ee147d8d914b89882bb72b57c29481. Latest consensus state: current_term: 1 leader_uuid: "24ee147d8d914b89882bb72b57c29481" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24ee147d8d914b89882bb72b57c29481" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 39245 } } }
I20240507 02:11:05.200964 18005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24ee147d8d914b89882bb72b57c29481 [sys.catalog]: This master's current role is: LEADER
I20240507 02:11:05.201119 18004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24ee147d8d914b89882bb72b57c29481 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "24ee147d8d914b89882bb72b57c29481" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24ee147d8d914b89882bb72b57c29481" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 39245 } } }
I20240507 02:11:05.201155 18012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 02:11:05.201169 18004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24ee147d8d914b89882bb72b57c29481 [sys.catalog]: This master's current role is: LEADER
I20240507 02:11:05.201586 18012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240507 02:11:05.203141 18012 catalog_manager.cc:1347] Generated new cluster ID: 04c89ee763e84bbaaf29703f87ced4fc
I20240507 02:11:05.203184 18012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
May 07 02:11:05 dist-test-slave-v0bg krb5kdc[17917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:11:05.208132 18002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:05.199089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37556 (local address 127.31.131.126:39245)
0507 02:11:05.199230 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:05.199239 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:05.199261 (+ 22us) server_negotiation.cc:407] Connection header received
0507 02:11:05.199364 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:05.199369 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:05.199414 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:05.199533 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:05.200460 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:05.201582 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:05.202507 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:05.202719 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:05.204953 (+ 2234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:05.204975 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:05.204975 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:11:05.204990 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:05.206859 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:05.207173 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:05.207175 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:05.207176 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:05.207229 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:05.207365 (+ 136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:05.207365 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:05.207366 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:05.207586 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:05.207719 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:05.207823 (+ 104us) server_negotiation.cc:299] Negotiation successful
0507 02:11:05.207936 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":80,"threads_started":1}
I20240507 02:11:05.215551 18012 catalog_manager.cc:1370] Generated new certificate authority record
I20240507 02:11:05.216208 18012 catalog_manager.cc:1504] Loading token signing keys...
I20240507 02:11:05.224522 18012 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 24ee147d8d914b89882bb72b57c29481: Generated new TSK 0
I20240507 02:11:05.224663 18012 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.131.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab.
Entry for principal kudu/127.31.131.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab.
I20240507 02:11:05.246835 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.65:0
--local_ip_for_outbound_sockets=127.31.131.65
--webserver_interface=127.31.131.65
--webserver_port=0
--tserver_master_addrs=127.31.131.126:39245
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab
--principal=kudu/127.31.131.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:41763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:11:05.319519 18024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:11:05.319689 18024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:11:05.319708 18024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:11:05.320940 18024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:11:05.320976 18024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:11:05.320998 18024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.65
I20240507 02:11:05.322517 18024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:41763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab
--principal=kudu/127.31.131.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.131.65
--webserver_port=0
--tserver_master_addrs=127.31.131.126:39245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.65
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:11:05.322716 18024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:11:05.322871 18024 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:11:05.326833 18029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:11:05.326825 18032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:11:05.326823 18030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:05.326953 18024 server_base.cc:1034] running on GCE node
I20240507 02:11:05.327200 18024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:11:05.327378 18024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:11:05.328500 18024 hybrid_clock.cc:648] HybridClock initialized: now 1715047865328487 us; error 29 us; skew 500 ppm
May 07 02:11:05 dist-test-slave-v0bg krb5kdc[17917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047865, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:11:05.330744 18024 init.cc:377] Logged in from keytab as kudu/127.31.131.65@KRBTEST.COM (short username kudu)
I20240507 02:11:05.331192 18024 webserver.cc:460] Webserver started at http://127.31.131.65:38403/ using document root <none> and password file <none>
I20240507 02:11:05.331370 18024 fs_manager.cc:359] Metadata directory not provided
I20240507 02:11:05.331415 18024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:11:05.331516 18024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:11:05.332619 18024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/minicluster-data/ts-0/data/instance:
uuid: "03c33f9a6237418b87211dca270b5651"
format_stamp: "Formatted at 2024-05-07 02:11:05 on dist-test-slave-v0bg"
server_key: "0e1f0e7220008f2616d23103f7fcccdc"
server_key_iv: "b64111171db0b39a557c79c7ff86f052"
server_key_version: "encryptionkey@0"
I20240507 02:11:05.332924 18024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/minicluster-data/ts-0/wal/instance:
uuid: "03c33f9a6237418b87211dca270b5651"
format_stamp: "Formatted at 2024-05-07 02:11:05 on dist-test-slave-v0bg"
server_key: "0e1f0e7220008f2616d23103f7fcccdc"
server_key_iv: "b64111171db0b39a557c79c7ff86f052"
server_key_version: "encryptionkey@0"
I20240507 02:11:05.334151 18024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240507 02:11:05.334854 18039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:05.335016 18024 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:05.335090 18024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/minicluster-data/ts-0/wal
uuid: "03c33f9a6237418b87211dca270b5651"
format_stamp: "Formatted at 2024-05-07 02:11:05 on dist-test-slave-v0bg"
server_key: "0e1f0e7220008f2616d23103f7fcccdc"
server_key_iv: "b64111171db0b39a557c79c7ff86f052"
server_key_version: "encryptionkey@0"
I20240507 02:11:05.335150 18024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:11:05.355998 18024 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:11:05.357309 18024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:11:05.357388 18024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:11:05.357532 18024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:11:05.357740 18024 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:11:05.357767 18024 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:05.357785 18024 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:11:05.357801 18024 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:05.362808 18024 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.65:38459
I20240507 02:11:05.362854 18152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.65:38459 every 8 connection(s)
I20240507 02:11:05.363087 18024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
May 07 02:11:05 dist-test-slave-v0bg krb5kdc[17917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047865, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.65@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:11:05.369786 18002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:05.363659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:41519 (local address 127.31.131.126:39245)
0507 02:11:05.363724 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:05.363726 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:05.363850 (+ 124us) server_negotiation.cc:407] Connection header received
0507 02:11:05.364254 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:05.364255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:05.364265 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:05.364308 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:05.365178 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:05.365645 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:05.366116 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:05.366216 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:05.367862 (+ 1646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:05.367865 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:05.367865 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:11:05.367871 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:05.369022 (+ 1151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:05.369272 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:05.369273 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:05.369273 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:05.369288 (+ 15us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:05.369398 (+ 110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:05.369398 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:05.369398 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:05.369468 (+ 70us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:05.369508 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:05.369670 (+ 162us) server_negotiation.cc:299] Negotiation successful
0507 02:11:05.369726 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20240507 02:11:05.369879 18155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:05.363634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:39245 (local address 127.31.131.65:41519)
0507 02:11:05.363761 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:05.363783 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0507 02:11:05.364183 (+ 400us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:11:05.364351 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:11:05.364355 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:11:05.364601 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:11:05.365116 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:05.365120 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:05.365703 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:11:05.365703 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:11:05.366068 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:05.366069 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:05.366153 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:05.366597 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:11:05.366615 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:11:05.367780 (+ 1165us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:11:05.369079 (+ 1299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:11:05.369081 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:11:05.369083 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:11:05.369226 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:11:05.369328 (+ 102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:11:05.369328 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:11:05.369329 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:11:05.369368 (+ 39us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:11:05.369507 (+ 139us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:11:05.369509 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:11:05.369591 (+ 82us) client_negotiation.cc:769] Sending connection context
0507 02:11:05.369662 (+ 71us) client_negotiation.cc:240] Negotiation successful
0507 02:11:05.369745 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":68,"threads_started":1}
I20240507 02:11:05.370224 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 18024
I20240507 02:11:05.370321 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/minicluster-data/ts-0/wal/instance
I20240507 02:11:05.370354 18153 heartbeater.cc:347] Connected to a master server at 127.31.131.126:39245
I20240507 02:11:05.370438 18153 heartbeater.cc:464] Registering TS with master...
I20240507 02:11:05.370509 32269 external_mini_cluster.cc:1428] Setting key 243524580a2aa50c3cf81b29ddd6e6f6
I20240507 02:11:05.370644 18153 heartbeater.cc:511] Master 127.31.131.126:39245 requested a full tablet report, sending...
I20240507 02:11:05.371079 17961 ts_manager.cc:194] Registered new tserver with Master: 03c33f9a6237418b87211dca270b5651 (127.31.131.65:38459)
I20240507 02:11:05.371723 17961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.131.65@KRBTEST.COM'} at 127.31.131.65:41519
WARNING: no policy specified for kudu/127.31.131.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab.
Entry for principal kudu/127.31.131.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab.
I20240507 02:11:05.390628 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.66:0
--local_ip_for_outbound_sockets=127.31.131.66
--webserver_interface=127.31.131.66
--webserver_port=0
--tserver_master_addrs=127.31.131.126:39245
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab
--principal=kudu/127.31.131.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:41763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:11:05.463613 18158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:11:05.463771 18158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:11:05.463791 18158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:11:05.465029 18158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:11:05.465070 18158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:11:05.465095 18158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.66
I20240507 02:11:05.466539 18158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:41763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab
--principal=kudu/127.31.131.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.131.66
--webserver_port=0
--tserver_master_addrs=127.31.131.126:39245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.66
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:11:05.466744 18158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:11:05.466902 18158 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:11:05.470340 18164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:05.470492 18158 server_base.cc:1034] running on GCE node
W20240507 02:11:05.470531 18163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:11:05.470383 18166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:05.470728 18158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:11:05.470911 18158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:11:05.472039 18158 hybrid_clock.cc:648] HybridClock initialized: now 1715047865472022 us; error 30 us; skew 500 ppm
May 07 02:11:05 dist-test-slave-v0bg krb5kdc[17917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047865, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:11:05.474210 18158 init.cc:377] Logged in from keytab as kudu/127.31.131.66@KRBTEST.COM (short username kudu)
I20240507 02:11:05.474601 18158 webserver.cc:460] Webserver started at http://127.31.131.66:44093/ using document root <none> and password file <none>
I20240507 02:11:05.474761 18158 fs_manager.cc:359] Metadata directory not provided
I20240507 02:11:05.474798 18158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:11:05.474900 18158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:11:05.475764 18158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/minicluster-data/ts-1/data/instance:
uuid: "79904e40bddf45c89d43c2588f285fef"
format_stamp: "Formatted at 2024-05-07 02:11:05 on dist-test-slave-v0bg"
server_key: "f5360187f87510450d141b57f7e66ce6"
server_key_iv: "c13b069728298bab689a6fb722903b98"
server_key_version: "encryptionkey@0"
I20240507 02:11:05.476033 18158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/minicluster-data/ts-1/wal/instance:
uuid: "79904e40bddf45c89d43c2588f285fef"
format_stamp: "Formatted at 2024-05-07 02:11:05 on dist-test-slave-v0bg"
server_key: "f5360187f87510450d141b57f7e66ce6"
server_key_iv: "c13b069728298bab689a6fb722903b98"
server_key_version: "encryptionkey@0"
I20240507 02:11:05.477011 18158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240507 02:11:05.477542 18173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:05.477682 18158 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240507 02:11:05.477747 18158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/minicluster-data/ts-1/wal
uuid: "79904e40bddf45c89d43c2588f285fef"
format_stamp: "Formatted at 2024-05-07 02:11:05 on dist-test-slave-v0bg"
server_key: "f5360187f87510450d141b57f7e66ce6"
server_key_iv: "c13b069728298bab689a6fb722903b98"
server_key_version: "encryptionkey@0"
I20240507 02:11:05.477800 18158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:11:05.496501 18158 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:11:05.497833 18158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:11:05.497928 18158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:11:05.498091 18158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:11:05.498337 18158 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:11:05.498366 18158 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:05.498392 18158 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:11:05.498415 18158 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:05.503355 18158 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.66:44763
I20240507 02:11:05.503410 18286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.66:44763 every 8 connection(s)
I20240507 02:11:05.503636 18158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
May 07 02:11:05 dist-test-slave-v0bg krb5kdc[17917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047865, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.66@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:11:05.510393 18002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:05.504185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.66:45233 (local address 127.31.131.126:39245)
0507 02:11:05.504241 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:05.504243 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:05.504370 (+ 127us) server_negotiation.cc:407] Connection header received
0507 02:11:05.504810 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:05.504812 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:05.504823 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:05.504873 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:05.505750 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:05.506228 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:05.506765 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:05.506925 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:05.508535 (+ 1610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:05.508538 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:05.508539 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:11:05.508545 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:05.509572 (+ 1027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:05.509858 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:05.509859 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:05.509859 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:05.509874 (+ 15us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:05.509994 (+ 120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:05.509994 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:05.509994 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:05.510067 (+ 73us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:05.510107 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:05.510279 (+ 172us) server_negotiation.cc:299] Negotiation successful
0507 02:11:05.510333 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20240507 02:11:05.510483 18289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:05.504155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:39245 (local address 127.31.131.66:45233)
0507 02:11:05.504277 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:05.504299 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0507 02:11:05.504711 (+ 412us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:11:05.504924 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:11:05.504927 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:11:05.505186 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:11:05.505657 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:05.505660 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:05.506303 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:11:05.506304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:11:05.506688 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:05.506689 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:05.506804 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:05.507227 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:11:05.507240 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:11:05.508441 (+ 1201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:11:05.509636 (+ 1195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:11:05.509640 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:11:05.509642 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:11:05.509806 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:11:05.509916 (+ 110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:11:05.509917 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:11:05.509917 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:11:05.509959 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:11:05.510104 (+ 145us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:11:05.510106 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:11:05.510194 (+ 88us) client_negotiation.cc:769] Sending connection context
0507 02:11:05.510268 (+ 74us) client_negotiation.cc:240] Negotiation successful
0507 02:11:05.510354 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":63,"threads_started":1}
I20240507 02:11:05.510890 18287 heartbeater.cc:347] Connected to a master server at 127.31.131.126:39245
I20240507 02:11:05.510970 18287 heartbeater.cc:464] Registering TS with master...
I20240507 02:11:05.511135 18287 heartbeater.cc:511] Master 127.31.131.126:39245 requested a full tablet report, sending...
I20240507 02:11:05.511495 17961 ts_manager.cc:194] Registered new tserver with Master: 79904e40bddf45c89d43c2588f285fef (127.31.131.66:44763)
I20240507 02:11:05.511948 17961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.131.66@KRBTEST.COM'} at 127.31.131.66:45233
I20240507 02:11:05.513757 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 18158
I20240507 02:11:05.513865 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/minicluster-data/ts-1/wal/instance
I20240507 02:11:05.514046 32269 external_mini_cluster.cc:1428] Setting key df1c2badd25f3a6f273e317dddcc46cc
WARNING: no policy specified for kudu/127.31.131.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab.
Entry for principal kudu/127.31.131.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab.
I20240507 02:11:05.533100 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.67:0
--local_ip_for_outbound_sockets=127.31.131.67
--webserver_interface=127.31.131.67
--webserver_port=0
--tserver_master_addrs=127.31.131.126:39245
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab
--principal=kudu/127.31.131.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:41763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:11:05.606779 18292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:11:05.606933 18292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:11:05.606952 18292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:11:05.608184 18292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:11:05.608222 18292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:11:05.608246 18292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.67
I20240507 02:11:05.609669 18292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:41763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab
--principal=kudu/127.31.131.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.131.67
--webserver_port=0
--tserver_master_addrs=127.31.131.126:39245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.67
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:11:05.609838 18292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:11:05.609990 18292 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:11:05.613373 18298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:05.613389 18292 server_base.cc:1034] running on GCE node
W20240507 02:11:05.613492 18297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:11:05.613373 18300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:05.613718 18292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:11:05.613894 18292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:11:05.615011 18292 hybrid_clock.cc:648] HybridClock initialized: now 1715047865614999 us; error 29 us; skew 500 ppm
May 07 02:11:05 dist-test-slave-v0bg krb5kdc[17917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047865, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:11:05.616972 18292 init.cc:377] Logged in from keytab as kudu/127.31.131.67@KRBTEST.COM (short username kudu)
I20240507 02:11:05.617326 18292 webserver.cc:460] Webserver started at http://127.31.131.67:34733/ using document root <none> and password file <none>
I20240507 02:11:05.617471 18292 fs_manager.cc:359] Metadata directory not provided
I20240507 02:11:05.617508 18292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:11:05.617614 18292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:11:05.618419 18292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/minicluster-data/ts-2/data/instance:
uuid: "3b1514ce1ead48b2a7c4a8079b384c1f"
format_stamp: "Formatted at 2024-05-07 02:11:05 on dist-test-slave-v0bg"
server_key: "6bca3d44818b90f50b52f372b29f6fc0"
server_key_iv: "40567631679d8996519fdb4ac8f35dd0"
server_key_version: "encryptionkey@0"
I20240507 02:11:05.618713 18292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/minicluster-data/ts-2/wal/instance:
uuid: "3b1514ce1ead48b2a7c4a8079b384c1f"
format_stamp: "Formatted at 2024-05-07 02:11:05 on dist-test-slave-v0bg"
server_key: "6bca3d44818b90f50b52f372b29f6fc0"
server_key_iv: "40567631679d8996519fdb4ac8f35dd0"
server_key_version: "encryptionkey@0"
I20240507 02:11:05.619735 18292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240507 02:11:05.620265 18307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:05.620398 18292 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240507 02:11:05.620466 18292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/minicluster-data/ts-2/wal
uuid: "3b1514ce1ead48b2a7c4a8079b384c1f"
format_stamp: "Formatted at 2024-05-07 02:11:05 on dist-test-slave-v0bg"
server_key: "6bca3d44818b90f50b52f372b29f6fc0"
server_key_iv: "40567631679d8996519fdb4ac8f35dd0"
server_key_version: "encryptionkey@0"
I20240507 02:11:05.620529 18292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:11:05.628304 18292 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:11:05.629563 18292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:11:05.629652 18292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:11:05.629794 18292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:11:05.630020 18292 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:11:05.630049 18292 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:05.630072 18292 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:11:05.630088 18292 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:05.634799 18292 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.67:41483
I20240507 02:11:05.634835 18420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.67:41483 every 8 connection(s)
I20240507 02:11:05.635056 18292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
I20240507 02:11:05.636747 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 18292
I20240507 02:11:05.636863 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/minicluster-data/ts-2/wal/instance
I20240507 02:11:05.637033 32269 external_mini_cluster.cc:1428] Setting key 41e0176eaba1badf2178d95898b545ea
May 07 02:11:05 dist-test-slave-v0bg krb5kdc[17917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047865, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.67@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:11:05.641973 18002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:05.635585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.67:47283 (local address 127.31.131.126:39245)
0507 02:11:05.635643 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:05.635645 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:05.635778 (+ 133us) server_negotiation.cc:407] Connection header received
0507 02:11:05.636161 (+ 383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:05.636162 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:05.636171 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:05.636216 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:05.637174 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:05.637857 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:05.638400 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:05.638509 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:05.640129 (+ 1620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:05.640131 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:05.640132 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:11:05.640137 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:05.641137 (+ 1000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:05.641412 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:05.641413 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:05.641413 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:05.641426 (+ 13us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:05.641551 (+ 125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:05.641551 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:05.641551 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:05.641618 (+ 67us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:05.641659 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:05.641856 (+ 197us) server_negotiation.cc:299] Negotiation successful
0507 02:11:05.641913 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20240507 02:11:05.642089 18423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:05.635532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:39245 (local address 127.31.131.67:47283)
0507 02:11:05.635674 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:05.635695 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0507 02:11:05.636083 (+ 388us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:11:05.636253 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:11:05.636256 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:11:05.636487 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:11:05.637088 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:05.637093 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:05.637920 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:11:05.637921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:11:05.638338 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:05.638340 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:05.638432 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:05.638843 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:11:05.638855 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:11:05.640056 (+ 1201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:11:05.641197 (+ 1141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:11:05.641200 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:11:05.641202 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:11:05.641361 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:11:05.641468 (+ 107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:11:05.641468 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:11:05.641468 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:11:05.641513 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:11:05.641656 (+ 143us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:11:05.641658 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:11:05.641740 (+ 82us) client_negotiation.cc:769] Sending connection context
0507 02:11:05.641843 (+ 103us) client_negotiation.cc:240] Negotiation successful
0507 02:11:05.641941 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":80,"threads_started":1}
I20240507 02:11:05.642473 18421 heartbeater.cc:347] Connected to a master server at 127.31.131.126:39245
I20240507 02:11:05.642570 18421 heartbeater.cc:464] Registering TS with master...
I20240507 02:11:05.642717 18421 heartbeater.cc:511] Master 127.31.131.126:39245 requested a full tablet report, sending...
I20240507 02:11:05.643015 17961 ts_manager.cc:194] Registered new tserver with Master: 3b1514ce1ead48b2a7c4a8079b384c1f (127.31.131.67:41483)
I20240507 02:11:05.643348 17961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.131.67@KRBTEST.COM'} at 127.31.131.67:47283
I20240507 02:11:05.648321 32269 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240507 02:11:05.653826 18002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:05.649429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37562 (local address 127.31.131.126:39245)
0507 02:11:05.649480 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:05.649482 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:05.649566 (+ 84us) server_negotiation.cc:407] Connection header received
0507 02:11:05.649602 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:05.649602 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:05.649609 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:05.649652 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:05.650216 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:05.650665 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:05.651143 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:05.651234 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:05.652120 (+ 886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:05.652123 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:05.652123 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:11:05.652126 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:05.653075 (+ 949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:05.653353 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:05.653354 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:05.653354 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:05.653368 (+ 14us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:05.653474 (+ 106us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:05.653474 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:05.653474 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:05.653560 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:05.653630 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:05.653721 (+ 91us) server_negotiation.cc:299] Negotiation successful
0507 02:11:05.653759 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20240507 02:11:05.655208 17961 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37562:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 02:11:05.656508 17961 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 02:11:05.660753 18429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:05.658098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.65:38459 (local address 127.0.0.1:33648)
0507 02:11:05.658345 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:05.658354 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0507 02:11:05.658400 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:11:05.658810 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:11:05.658810 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:11:05.658812 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:11:05.658995 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:05.658997 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:05.659956 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:11:05.659957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:11:05.660525 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:05.660526 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:05.660582 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:05.660609 (+ 27us) client_negotiation.cc:769] Sending connection context
0507 02:11:05.660651 (+ 42us) client_negotiation.cc:240] Negotiation successful
0507 02:11:05.660689 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":78,"threads_started":1}
I20240507 02:11:05.660959 18430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:05.658083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.66:44763 (local address 127.0.0.1:41764)
0507 02:11:05.658234 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:05.658256 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0507 02:11:05.658318 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:11:05.658595 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:11:05.658598 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:11:05.658599 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:11:05.658807 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:05.658810 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:05.659977 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:11:05.659978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:11:05.660767 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:05.660769 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:05.660826 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:05.660829 (+ 3us) client_negotiation.cc:769] Sending connection context
0507 02:11:05.660859 (+ 30us) client_negotiation.cc:240] Negotiation successful
0507 02:11:05.660896 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":46,"threads_started":1}
I20240507 02:11:05.661598 18433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:05.658246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33648 (local address 127.31.131.65:38459)
0507 02:11:05.658655 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:05.658662 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:05.658672 (+ 10us) server_negotiation.cc:407] Connection header received
0507 02:11:05.658707 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:05.658709 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:05.658723 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:05.658800 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:11:05.659068 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:05.659886 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:05.660929 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:05.661405 (+ 476us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:05.661446 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:05.661475 (+ 29us) server_negotiation.cc:299] Negotiation successful
0507 02:11:05.661521 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":77,"threads_started":1}
I20240507 02:11:05.661720 18432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:05.658254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.67:41483 (local address 127.0.0.1:43438)
0507 02:11:05.658510 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:05.658518 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0507 02:11:05.658873 (+ 355us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:11:05.659192 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:11:05.659192 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:11:05.659195 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:11:05.659387 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:05.659388 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:05.660587 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:11:05.660588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:11:05.661531 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:05.661534 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:05.661603 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:05.661606 (+ 3us) client_negotiation.cc:769] Sending connection context
0507 02:11:05.661634 (+ 28us) client_negotiation.cc:240] Negotiation successful
0507 02:11:05.661666 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":52,"threads_started":1}
I20240507 02:11:05.661979 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:05.658202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41764 (local address 127.31.131.66:44763)
0507 02:11:05.658417 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:05.658423 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:05.658430 (+ 7us) server_negotiation.cc:407] Connection header received
0507 02:11:05.658458 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:05.658460 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:05.658475 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:05.658573 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:11:05.658881 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:05.659888 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:05.661246 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:05.661583 (+ 337us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:05.661624 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:05.661858 (+ 234us) server_negotiation.cc:299] Negotiation successful
0507 02:11:05.661907 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":76,"threads_started":1}
I20240507 02:11:05.662076 18087 tablet_service.cc:1467] Processing CreateTablet for tablet e2cdf60ef9f94b6587247b5e0b91a027 (DEFAULT_TABLE table=test-table [id=31a84caa69664e96ba814cd4db16f254]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:11:05.662226 18434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:05.658392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43438 (local address 127.31.131.67:41483)
0507 02:11:05.658570 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:05.658577 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:05.658586 (+ 9us) server_negotiation.cc:407] Connection header received
0507 02:11:05.658936 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:05.658938 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:05.658951 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:05.659143 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:11:05.659458 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:05.660509 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:05.661628 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:05.662075 (+ 447us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:05.662112 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:05.662136 (+ 24us) server_negotiation.cc:299] Negotiation successful
0507 02:11:05.662169 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":67,"threads_started":1}
I20240507 02:11:05.662346 18087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2cdf60ef9f94b6587247b5e0b91a027. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:11:05.662451 18221 tablet_service.cc:1467] Processing CreateTablet for tablet e2cdf60ef9f94b6587247b5e0b91a027 (DEFAULT_TABLE table=test-table [id=31a84caa69664e96ba814cd4db16f254]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:11:05.662685 18221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2cdf60ef9f94b6587247b5e0b91a027. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:11:05.662679 18355 tablet_service.cc:1467] Processing CreateTablet for tablet e2cdf60ef9f94b6587247b5e0b91a027 (DEFAULT_TABLE table=test-table [id=31a84caa69664e96ba814cd4db16f254]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:11:05.662896 18355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2cdf60ef9f94b6587247b5e0b91a027. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:11:05.664786 18435 tablet_bootstrap.cc:492] T e2cdf60ef9f94b6587247b5e0b91a027 P 79904e40bddf45c89d43c2588f285fef: Bootstrap starting.
I20240507 02:11:05.665175 18436 tablet_bootstrap.cc:492] T e2cdf60ef9f94b6587247b5e0b91a027 P 03c33f9a6237418b87211dca270b5651: Bootstrap starting.
I20240507 02:11:05.665254 18435 tablet_bootstrap.cc:654] T e2cdf60ef9f94b6587247b5e0b91a027 P 79904e40bddf45c89d43c2588f285fef: Neither blocks nor log segments found. Creating new log.
I20240507 02:11:05.665378 18437 tablet_bootstrap.cc:492] T e2cdf60ef9f94b6587247b5e0b91a027 P 3b1514ce1ead48b2a7c4a8079b384c1f: Bootstrap starting.
I20240507 02:11:05.665508 18435 log.cc:824] T e2cdf60ef9f94b6587247b5e0b91a027 P 79904e40bddf45c89d43c2588f285fef: Log is configured to *not* fsync() on all Append() calls
I20240507 02:11:05.665663 18436 tablet_bootstrap.cc:654] T e2cdf60ef9f94b6587247b5e0b91a027 P 03c33f9a6237418b87211dca270b5651: Neither blocks nor log segments found. Creating new log.
I20240507 02:11:05.665846 18437 tablet_bootstrap.cc:654] T e2cdf60ef9f94b6587247b5e0b91a027 P 3b1514ce1ead48b2a7c4a8079b384c1f: Neither blocks nor log segments found. Creating new log.
I20240507 02:11:05.665879 18436 log.cc:824] T e2cdf60ef9f94b6587247b5e0b91a027 P 03c33f9a6237418b87211dca270b5651: Log is configured to *not* fsync() on all Append() calls
I20240507 02:11:05.666121 18437 log.cc:824] T e2cdf60ef9f94b6587247b5e0b91a027 P 3b1514ce1ead48b2a7c4a8079b384c1f: Log is configured to *not* fsync() on all Append() calls
I20240507 02:11:05.666203 18435 tablet_bootstrap.cc:492] T e2cdf60ef9f94b6587247b5e0b91a027 P 79904e40bddf45c89d43c2588f285fef: No bootstrap required, opened a new log
I20240507 02:11:05.666271 18435 ts_tablet_manager.cc:1367] T e2cdf60ef9f94b6587247b5e0b91a027 P 79904e40bddf45c89d43c2588f285fef: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240507 02:11:05.666659 18436 tablet_bootstrap.cc:492] T e2cdf60ef9f94b6587247b5e0b91a027 P 03c33f9a6237418b87211dca270b5651: No bootstrap required, opened a new log
I20240507 02:11:05.666724 18436 ts_tablet_manager.cc:1367] T e2cdf60ef9f94b6587247b5e0b91a027 P 03c33f9a6237418b87211dca270b5651: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240507 02:11:05.667001 18437 tablet_bootstrap.cc:492] T e2cdf60ef9f94b6587247b5e0b91a027 P 3b1514ce1ead48b2a7c4a8079b384c1f: No bootstrap required, opened a new log
I20240507 02:11:05.667066 18437 ts_tablet_manager.cc:1367] T e2cdf60ef9f94b6587247b5e0b91a027 P 3b1514ce1ead48b2a7c4a8079b384c1f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240507 02:11:05.667866 18435 raft_consensus.cc:348] T e2cdf60ef9f94b6587247b5e0b91a027 P 79904e40bddf45c89d43c2588f285fef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79904e40bddf45c89d43c2588f285fef" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44763 } } peers { permanent_uuid: "3b1514ce1ead48b2a7c4a8079b384c1f" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 41483 } } peers { permanent_uuid: "03c33f9a6237418b87211dca270b5651" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 38459 } }
I20240507 02:11:05.667964 18435 raft_consensus.cc:374] T e2cdf60ef9f94b6587247b5e0b91a027 P 79904e40bddf45c89d43c2588f285fef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:11:05.667986 18435 raft_consensus.cc:729] T e2cdf60ef9f94b6587247b5e0b91a027 P 79904e40bddf45c89d43c2588f285fef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79904e40bddf45c89d43c2588f285fef, State: Initialized, Role: FOLLOWER
I20240507 02:11:05.668071 18435 consensus_queue.cc:260] T e2cdf60ef9f94b6587247b5e0b91a027 P 79904e40bddf45c89d43c2588f285fef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79904e40bddf45c89d43c2588f285fef" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44763 } } peers { permanent_uuid: "3b1514ce1ead48b2a7c4a8079b384c1f" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 41483 } } peers { permanent_uuid: "03c33f9a6237418b87211dca270b5651" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 38459 } }
I20240507 02:11:05.668277 18435 ts_tablet_manager.cc:1398] T e2cdf60ef9f94b6587247b5e0b91a027 P 79904e40bddf45c89d43c2588f285fef: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240507 02:11:05.668380 18287 heartbeater.cc:503] Master 127.31.131.126:39245 was elected leader, sending a full tablet report...
I20240507 02:11:05.668821 18436 raft_consensus.cc:348] T e2cdf60ef9f94b6587247b5e0b91a027 P 03c33f9a6237418b87211dca270b5651 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79904e40bddf45c89d43c2588f285fef" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44763 } } peers { permanent_uuid: "3b1514ce1ead48b2a7c4a8079b384c1f" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 41483 } } peers { permanent_uuid: "03c33f9a6237418b87211dca270b5651" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 38459 } }
I20240507 02:11:05.668951 18436 raft_consensus.cc:374] T e2cdf60ef9f94b6587247b5e0b91a027 P 03c33f9a6237418b87211dca270b5651 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:11:05.668980 18436 raft_consensus.cc:729] T e2cdf60ef9f94b6587247b5e0b91a027 P 03c33f9a6237418b87211dca270b5651 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03c33f9a6237418b87211dca270b5651, State: Initialized, Role: FOLLOWER
I20240507 02:11:05.669067 18437 raft_consensus.cc:348] T e2cdf60ef9f94b6587247b5e0b91a027 P 3b1514ce1ead48b2a7c4a8079b384c1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79904e40bddf45c89d43c2588f285fef" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44763 } } peers { permanent_uuid: "3b1514ce1ead48b2a7c4a8079b384c1f" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 41483 } } peers { permanent_uuid: "03c33f9a6237418b87211dca270b5651" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 38459 } }
I20240507 02:11:05.669170 18437 raft_consensus.cc:374] T e2cdf60ef9f94b6587247b5e0b91a027 P 3b1514ce1ead48b2a7c4a8079b384c1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:11:05.669196 18437 raft_consensus.cc:729] T e2cdf60ef9f94b6587247b5e0b91a027 P 3b1514ce1ead48b2a7c4a8079b384c1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b1514ce1ead48b2a7c4a8079b384c1f, State: Initialized, Role: FOLLOWER
I20240507 02:11:05.669153 18436 consensus_queue.cc:260] T e2cdf60ef9f94b6587247b5e0b91a027 P 03c33f9a6237418b87211dca270b5651 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79904e40bddf45c89d43c2588f285fef" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44763 } } peers { permanent_uuid: "3b1514ce1ead48b2a7c4a8079b384c1f" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 41483 } } peers { permanent_uuid: "03c33f9a6237418b87211dca270b5651" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 38459 } }
I20240507 02:11:05.669266 18437 consensus_queue.cc:260] T e2cdf60ef9f94b6587247b5e0b91a027 P 3b1514ce1ead48b2a7c4a8079b384c1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79904e40bddf45c89d43c2588f285fef" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44763 } } peers { permanent_uuid: "3b1514ce1ead48b2a7c4a8079b384c1f" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 41483 } } peers { permanent_uuid: "03c33f9a6237418b87211dca270b5651" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 38459 } }
I20240507 02:11:05.669363 18436 ts_tablet_manager.cc:1398] T e2cdf60ef9f94b6587247b5e0b91a027 P 03c33f9a6237418b87211dca270b5651: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240507 02:11:05.669404 18153 heartbeater.cc:503] Master 127.31.131.126:39245 was elected leader, sending a full tablet report...
I20240507 02:11:05.669464 18437 ts_tablet_manager.cc:1398] T e2cdf60ef9f94b6587247b5e0b91a027 P 3b1514ce1ead48b2a7c4a8079b384c1f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240507 02:11:05.669494 18421 heartbeater.cc:503] Master 127.31.131.126:39245 was elected leader, sending a full tablet report...
I20240507 02:11:05.744621 18443 raft_consensus.cc:482] T e2cdf60ef9f94b6587247b5e0b91a027 P 3b1514ce1ead48b2a7c4a8079b384c1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 02:11:05.744696 18443 raft_consensus.cc:504] T e2cdf60ef9f94b6587247b5e0b91a027 P 3b1514ce1ead48b2a7c4a8079b384c1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79904e40bddf45c89d43c2588f285fef" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44763 } } peers { permanent_uuid: "3b1514ce1ead48b2a7c4a8079b384c1f" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 41483 } } peers { permanent_uuid: "03c33f9a6237418b87211dca270b5651" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 38459 } }
I20240507 02:11:05.744935 18443 leader_election.cc:288] T e2cdf60ef9f94b6587247b5e0b91a027 P 3b1514ce1ead48b2a7c4a8079b384c1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79904e40bddf45c89d43c2588f285fef (127.31.131.66:44763), 03c33f9a6237418b87211dca270b5651 (127.31.131.65:38459)
I20240507 02:11:05.746752 18423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:05.745052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.66:44763 (local address 127.31.131.67:43405)
0507 02:11:05.745103 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:05.745113 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0507 02:11:05.745151 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:11:05.745289 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:11:05.745290 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:11:05.745291 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:11:05.745446 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:05.745447 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:05.746027 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:11:05.746028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:11:05.746574 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:05.746575 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:05.746632 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:05.746635 (+ 3us) client_negotiation.cc:769] Sending connection context
0507 02:11:05.746661 (+ 26us) client_negotiation.cc:240] Negotiation successful
0507 02:11:05.746693 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20240507 02:11:05.746920 18444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:05.745063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.65:38459 (local address 127.31.131.67:59223)
0507 02:11:05.745205 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:05.745214 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0507 02:11:05.745274 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:11:05.745374 (+ 100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:11:05.745375 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:11:05.745376 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:11:05.745572 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:05.745573 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:05.746149 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:11:05.746150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:11:05.746738 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:05.746739 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:05.746797 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:05.746800 (+ 3us) client_negotiation.cc:769] Sending connection context
0507 02:11:05.746825 (+ 25us) client_negotiation.cc:240] Negotiation successful
0507 02:11:05.746856 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":66,"threads_started":1}
I20240507 02:11:05.747174 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:05.745166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.67:43405 (local address 127.31.131.66:44763)
0507 02:11:05.745202 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:05.745203 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:05.745210 (+ 7us) server_negotiation.cc:407] Connection header received
0507 02:11:05.745228 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:05.745228 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:05.745235 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:05.745274 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:11:05.745506 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:05.745952 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:05.746641 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:05.747076 (+ 435us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:05.747084 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:05.747103 (+ 19us) server_negotiation.cc:299] Negotiation successful
0507 02:11:05.747129 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20240507 02:11:05.747298 18433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:05.745167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.67:59223 (local address 127.31.131.65:38459)
0507 02:11:05.745206 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:05.745207 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:05.745273 (+ 66us) server_negotiation.cc:407] Connection header received
0507 02:11:05.745319 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:05.745320 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:05.745328 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:05.745371 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:11:05.745634 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:05.746080 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:05.746803 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:05.747192 (+ 389us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:05.747198 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:05.747215 (+ 17us) server_negotiation.cc:299] Negotiation successful
0507 02:11:05.747241 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20240507 02:11:05.747434 18241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2cdf60ef9f94b6587247b5e0b91a027" candidate_uuid: "3b1514ce1ead48b2a7c4a8079b384c1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79904e40bddf45c89d43c2588f285fef" is_pre_election: true
I20240507 02:11:05.747524 18241 raft_consensus.cc:2454] T e2cdf60ef9f94b6587247b5e0b91a027 P 79904e40bddf45c89d43c2588f285fef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b1514ce1ead48b2a7c4a8079b384c1f in term 0.
I20240507 02:11:05.747537 18107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2cdf60ef9f94b6587247b5e0b91a027" candidate_uuid: "3b1514ce1ead48b2a7c4a8079b384c1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03c33f9a6237418b87211dca270b5651" is_pre_election: true
I20240507 02:11:05.747632 18107 raft_consensus.cc:2454] T e2cdf60ef9f94b6587247b5e0b91a027 P 03c33f9a6237418b87211dca270b5651 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b1514ce1ead48b2a7c4a8079b384c1f in term 0.
I20240507 02:11:05.747715 18311 leader_election.cc:302] T e2cdf60ef9f94b6587247b5e0b91a027 P 3b1514ce1ead48b2a7c4a8079b384c1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b1514ce1ead48b2a7c4a8079b384c1f, 79904e40bddf45c89d43c2588f285fef; no voters:
I20240507 02:11:05.747821 18443 raft_consensus.cc:2781] T e2cdf60ef9f94b6587247b5e0b91a027 P 3b1514ce1ead48b2a7c4a8079b384c1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 02:11:05.747879 18443 raft_consensus.cc:482] T e2cdf60ef9f94b6587247b5e0b91a027 P 3b1514ce1ead48b2a7c4a8079b384c1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 02:11:05.747898 18443 raft_consensus.cc:3037] T e2cdf60ef9f94b6587247b5e0b91a027 P 3b1514ce1ead48b2a7c4a8079b384c1f [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:11:05.748481 18443 raft_consensus.cc:504] T e2cdf60ef9f94b6587247b5e0b91a027 P 3b1514ce1ead48b2a7c4a8079b384c1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79904e40bddf45c89d43c2588f285fef" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44763 } } peers { permanent_uuid: "3b1514ce1ead48b2a7c4a8079b384c1f" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 41483 } } peers { permanent_uuid: "03c33f9a6237418b87211dca270b5651" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 38459 } }
I20240507 02:11:05.748710 18107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2cdf60ef9f94b6587247b5e0b91a027" candidate_uuid: "3b1514ce1ead48b2a7c4a8079b384c1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03c33f9a6237418b87211dca270b5651"
I20240507 02:11:05.748713 18241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2cdf60ef9f94b6587247b5e0b91a027" candidate_uuid: "3b1514ce1ead48b2a7c4a8079b384c1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79904e40bddf45c89d43c2588f285fef"
I20240507 02:11:05.748778 18241 raft_consensus.cc:3037] T e2cdf60ef9f94b6587247b5e0b91a027 P 79904e40bddf45c89d43c2588f285fef [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:11:05.748780 18107 raft_consensus.cc:3037] T e2cdf60ef9f94b6587247b5e0b91a027 P 03c33f9a6237418b87211dca270b5651 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:11:05.749349 18107 raft_consensus.cc:2454] T e2cdf60ef9f94b6587247b5e0b91a027 P 03c33f9a6237418b87211dca270b5651 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b1514ce1ead48b2a7c4a8079b384c1f in term 1.
I20240507 02:11:05.749361 18241 raft_consensus.cc:2454] T e2cdf60ef9f94b6587247b5e0b91a027 P 79904e40bddf45c89d43c2588f285fef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b1514ce1ead48b2a7c4a8079b384c1f in term 1.
I20240507 02:11:05.749513 18309 leader_election.cc:302] T e2cdf60ef9f94b6587247b5e0b91a027 P 3b1514ce1ead48b2a7c4a8079b384c1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03c33f9a6237418b87211dca270b5651, 3b1514ce1ead48b2a7c4a8079b384c1f; no voters:
I20240507 02:11:05.749662 18443 leader_election.cc:288] T e2cdf60ef9f94b6587247b5e0b91a027 P 3b1514ce1ead48b2a7c4a8079b384c1f [CANDIDATE]: Term 1 election: Requested vote from peers 79904e40bddf45c89d43c2588f285fef (127.31.131.66:44763), 03c33f9a6237418b87211dca270b5651 (127.31.131.65:38459)
I20240507 02:11:05.749683 18445 raft_consensus.cc:2781] T e2cdf60ef9f94b6587247b5e0b91a027 P 3b1514ce1ead48b2a7c4a8079b384c1f [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:11:05.749737 18445 raft_consensus.cc:686] T e2cdf60ef9f94b6587247b5e0b91a027 P 3b1514ce1ead48b2a7c4a8079b384c1f [term 1 LEADER]: Becoming Leader. State: Replica: 3b1514ce1ead48b2a7c4a8079b384c1f, State: Running, Role: LEADER
I20240507 02:11:05.749809 18445 consensus_queue.cc:237] T e2cdf60ef9f94b6587247b5e0b91a027 P 3b1514ce1ead48b2a7c4a8079b384c1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79904e40bddf45c89d43c2588f285fef" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44763 } } peers { permanent_uuid: "3b1514ce1ead48b2a7c4a8079b384c1f" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 41483 } } peers { permanent_uuid: "03c33f9a6237418b87211dca270b5651" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 38459 } }
I20240507 02:11:05.750423 17961 catalog_manager.cc:5484] T e2cdf60ef9f94b6587247b5e0b91a027 P 3b1514ce1ead48b2a7c4a8079b384c1f reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b1514ce1ead48b2a7c4a8079b384c1f (127.31.131.67). New cstate: current_term: 1 leader_uuid: "3b1514ce1ead48b2a7c4a8079b384c1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79904e40bddf45c89d43c2588f285fef" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44763 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b1514ce1ead48b2a7c4a8079b384c1f" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 41483 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03c33f9a6237418b87211dca270b5651" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 38459 } health_report { overall_health: UNKNOWN } } }
W20240507 02:11:05.754026 18288 tablet.cc:2369] T e2cdf60ef9f94b6587247b5e0b91a027 P 79904e40bddf45c89d43c2588f285fef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240507 02:11:05.775120 18434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:05.773136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43442 (local address 127.31.131.67:41483)
0507 02:11:05.773177 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:05.773179 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:05.773188 (+ 9us) server_negotiation.cc:407] Connection header received
0507 02:11:05.773264 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:05.773265 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:05.773279 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:05.773333 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0507 02:11:05.773550 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:05.773982 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:05.774534 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:05.774696 (+ 162us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:05.774822 (+ 126us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0507 02:11:05.774898 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0507 02:11:05.774970 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:05.775055 (+ 85us) server_negotiation.cc:299] Negotiation successful
0507 02:11:05.775062 (+ 7us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20240507 02:11:05.776368 18241 raft_consensus.cc:1261] T e2cdf60ef9f94b6587247b5e0b91a027 P 79904e40bddf45c89d43c2588f285fef [term 1 FOLLOWER]: Refusing update from remote peer 3b1514ce1ead48b2a7c4a8079b384c1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 02:11:05.776403 18107 raft_consensus.cc:1261] T e2cdf60ef9f94b6587247b5e0b91a027 P 03c33f9a6237418b87211dca270b5651 [term 1 FOLLOWER]: Refusing update from remote peer 3b1514ce1ead48b2a7c4a8079b384c1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 02:11:05.776670 18443 consensus_queue.cc:1035] T e2cdf60ef9f94b6587247b5e0b91a027 P 3b1514ce1ead48b2a7c4a8079b384c1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "79904e40bddf45c89d43c2588f285fef" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 44763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 02:11:05.776777 18445 consensus_queue.cc:1035] T e2cdf60ef9f94b6587247b5e0b91a027 P 3b1514ce1ead48b2a7c4a8079b384c1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "03c33f9a6237418b87211dca270b5651" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 38459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 02:11:05.779556 18450 mvcc.cc:205] Tried to move back new op lower bound from 7024836058218188800 to 7024836058111750144. Current Snapshot: MvccSnapshot[applied={T|T < 7024836058218188800 or (T in {7024836058218188800})}]
I20240507 02:11:05.780779 17962 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37562:
table { table_name: "test-table" } modify_external_catalogs: true
I20240507 02:11:05.780866 17962 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37562:
table { table_name: "test-table" } modify_external_catalogs: true
I20240507 02:11:05.781488 17962 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 24ee147d8d914b89882bb72b57c29481: Sending DeleteTablet for 3 replicas of tablet e2cdf60ef9f94b6587247b5e0b91a027
I20240507 02:11:05.781800 18355 tablet_service.cc:1514] Processing DeleteTablet for tablet e2cdf60ef9f94b6587247b5e0b91a027 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:11:05 UTC) from {username='kudu', principal='kudu/127.31.131.126@KRBTEST.COM'} at 127.0.0.1:43438
I20240507 02:11:05.781834 18087 tablet_service.cc:1514] Processing DeleteTablet for tablet e2cdf60ef9f94b6587247b5e0b91a027 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:11:05 UTC) from {username='kudu', principal='kudu/127.31.131.126@KRBTEST.COM'} at 127.0.0.1:33648
I20240507 02:11:05.782011 18459 tablet_replica.cc:330] T e2cdf60ef9f94b6587247b5e0b91a027 P 03c33f9a6237418b87211dca270b5651: stopping tablet replica
I20240507 02:11:05.782043 18221 tablet_service.cc:1514] Processing DeleteTablet for tablet e2cdf60ef9f94b6587247b5e0b91a027 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:11:05 UTC) from {username='kudu', principal='kudu/127.31.131.126@KRBTEST.COM'} at 127.0.0.1:41764
I20240507 02:11:05.782076 18459 raft_consensus.cc:2229] T e2cdf60ef9f94b6587247b5e0b91a027 P 03c33f9a6237418b87211dca270b5651 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 02:11:05.782151 18459 raft_consensus.cc:2258] T e2cdf60ef9f94b6587247b5e0b91a027 P 03c33f9a6237418b87211dca270b5651 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 02:11:05.782230 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 18024
W20240507 02:11:05.787492 17947 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.131.65:38459 (error 108)
I20240507 02:11:05.787583 18458 tablet_replica.cc:330] T e2cdf60ef9f94b6587247b5e0b91a027 P 3b1514ce1ead48b2a7c4a8079b384c1f: stopping tablet replica
I20240507 02:11:05.787668 18458 raft_consensus.cc:2229] T e2cdf60ef9f94b6587247b5e0b91a027 P 3b1514ce1ead48b2a7c4a8079b384c1f [term 1 LEADER]: Raft consensus shutting down.
I20240507 02:11:05.787750 18458 raft_consensus.cc:2258] T e2cdf60ef9f94b6587247b5e0b91a027 P 3b1514ce1ead48b2a7c4a8079b384c1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 02:11:05.787878 18460 tablet_replica.cc:330] T e2cdf60ef9f94b6587247b5e0b91a027 P 79904e40bddf45c89d43c2588f285fef: stopping tablet replica
I20240507 02:11:05.787945 18460 raft_consensus.cc:2229] T e2cdf60ef9f94b6587247b5e0b91a027 P 79904e40bddf45c89d43c2588f285fef [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 02:11:05.787954 18432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:05.787841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.65:38459 (local address 127.0.0.1:33652)
0507 02:11:05.787894 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:05.787927 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.65:38459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20240507 02:11:05.788026 18460 raft_consensus.cc:2258] T e2cdf60ef9f94b6587247b5e0b91a027 P 79904e40bddf45c89d43c2588f285fef [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 02:11:05.788159 18458 ts_tablet_manager.cc:1875] T e2cdf60ef9f94b6587247b5e0b91a027 P 3b1514ce1ead48b2a7c4a8079b384c1f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240507 02:11:05.788167 17947 catalog_manager.cc:4573] TS 03c33f9a6237418b87211dca270b5651 (127.31.131.65:38459): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e2cdf60ef9f94b6587247b5e0b91a027: Network error: Client connection negotiation failed: client connection to 127.31.131.65:38459: connect: Connection refused (error 111)
I20240507 02:11:05.788374 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 18158
I20240507 02:11:05.788403 18460 ts_tablet_manager.cc:1875] T e2cdf60ef9f94b6587247b5e0b91a027 P 79904e40bddf45c89d43c2588f285fef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 02:11:05.790349 18458 ts_tablet_manager.cc:1888] T e2cdf60ef9f94b6587247b5e0b91a027 P 3b1514ce1ead48b2a7c4a8079b384c1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240507 02:11:05.790391 18458 log.cc:1196] T e2cdf60ef9f94b6587247b5e0b91a027 P 3b1514ce1ead48b2a7c4a8079b384c1f: Deleting WAL directory at /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1715047604443657-32269-0/minicluster-data/ts-2/wal/wals/e2cdf60ef9f94b6587247b5e0b91a027
I20240507 02:11:05.790602 18458 ts_tablet_manager.cc:1909] T e2cdf60ef9f94b6587247b5e0b91a027 P 3b1514ce1ead48b2a7c4a8079b384c1f: Deleting consensus metadata
I20240507 02:11:05.790912 17949 catalog_manager.cc:4830] TS 3b1514ce1ead48b2a7c4a8079b384c1f (127.31.131.67:41483): tablet e2cdf60ef9f94b6587247b5e0b91a027 (table test-table [id=31a84caa69664e96ba814cd4db16f254]) successfully deleted
I20240507 02:11:05.793313 18432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:05.793231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.66:44763 (local address 127.0.0.1:41774)
0507 02:11:05.793270 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:05.793291 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.66:44763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20240507 02:11:05.793315 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 18292
I20240507 02:11:05.797161 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 17930
2024-05-07T02:11:05Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1692 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2024-05-07T02:11:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-07T02:11:05Z Disabled control of system clock
I20240507 02:11:05.820556 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715047604443657-32269-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715047604443657-32269-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715047604443657-32269-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:45119
--webserver_interface=127.31.131.126
--webserver_port=0
--builtin_ntp_servers=127.31.131.84:41833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.131.126:45119
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20240507 02:11:05.895480 18465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:11:05.895656 18465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:11:05.895681 18465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:11:05.896888 18465 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240507 02:11:05.896921 18465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:11:05.896932 18465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:11:05.896942 18465 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240507 02:11:05.896965 18465 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240507 02:11:05.898280 18465 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:41833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715047604443657-32269-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715047604443657-32269-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.131.126:45119
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:45119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.131.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715047604443657-32269-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:11:05.898429 18465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:11:05.898645 18465 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:11:05.901988 18471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:05.902130 18465 server_base.cc:1034] running on GCE node
W20240507 02:11:05.901988 18473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:11:05.902002 18470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:05.902427 18465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:11:05.902698 18465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:11:05.903825 18465 hybrid_clock.cc:648] HybridClock initialized: now 1715047865903807 us; error 34 us; skew 500 ppm
I20240507 02:11:05.904839 18465 webserver.cc:460] Webserver started at http://127.31.131.126:33231/ using document root <none> and password file <none>
I20240507 02:11:05.904989 18465 fs_manager.cc:359] Metadata directory not provided
I20240507 02:11:05.905025 18465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:11:05.905108 18465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:11:05.905983 18465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715047604443657-32269-0/minicluster-data/master-0/data/instance:
uuid: "f7a333f8e08c464fa242d908852a172c"
format_stamp: "Formatted at 2024-05-07 02:11:05 on dist-test-slave-v0bg"
server_key: "9b11b89334bb9d21b29c76170f2a7346"
server_key_iv: "8fbd3a0de31c3388f39da8a99f062791"
server_key_version: "encryptionkey@0"
I20240507 02:11:05.906240 18465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715047604443657-32269-0/minicluster-data/master-0/wal/instance:
uuid: "f7a333f8e08c464fa242d908852a172c"
format_stamp: "Formatted at 2024-05-07 02:11:05 on dist-test-slave-v0bg"
server_key: "9b11b89334bb9d21b29c76170f2a7346"
server_key_iv: "8fbd3a0de31c3388f39da8a99f062791"
server_key_version: "encryptionkey@0"
I20240507 02:11:05.907306 18465 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240507 02:11:05.907899 18479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:05.908054 18465 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240507 02:11:05.908129 18465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715047604443657-32269-0/minicluster-data/master-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715047604443657-32269-0/minicluster-data/master-0/wal
uuid: "f7a333f8e08c464fa242d908852a172c"
format_stamp: "Formatted at 2024-05-07 02:11:05 on dist-test-slave-v0bg"
server_key: "9b11b89334bb9d21b29c76170f2a7346"
server_key_iv: "8fbd3a0de31c3388f39da8a99f062791"
server_key_version: "encryptionkey@0"
I20240507 02:11:05.908180 18465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715047604443657-32269-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715047604443657-32269-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715047604443657-32269-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:11:05.921825 18465 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:11:05.921998 18465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:11:05.922068 18465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:11:05.925458 18465 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.126:45119
I20240507 02:11:05.925484 18531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.126:45119 every 8 connection(s)
I20240507 02:11:05.925726 18465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
I20240507 02:11:05.926213 18532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:11:05.928323 18532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7a333f8e08c464fa242d908852a172c: Bootstrap starting.
I20240507 02:11:05.928756 18532 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7a333f8e08c464fa242d908852a172c: Neither blocks nor log segments found. Creating new log.
I20240507 02:11:05.928946 18532 log.cc:824] T 00000000000000000000000000000000 P f7a333f8e08c464fa242d908852a172c: Log is configured to *not* fsync() on all Append() calls
I20240507 02:11:05.929450 18532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7a333f8e08c464fa242d908852a172c: No bootstrap required, opened a new log
I20240507 02:11:05.930761 18532 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7a333f8e08c464fa242d908852a172c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7a333f8e08c464fa242d908852a172c" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 45119 } }
I20240507 02:11:05.930862 18532 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7a333f8e08c464fa242d908852a172c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:11:05.930877 18532 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7a333f8e08c464fa242d908852a172c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7a333f8e08c464fa242d908852a172c, State: Initialized, Role: FOLLOWER
I20240507 02:11:05.930961 18532 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7a333f8e08c464fa242d908852a172c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7a333f8e08c464fa242d908852a172c" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 45119 } }
I20240507 02:11:05.931003 18532 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7a333f8e08c464fa242d908852a172c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 02:11:05.931020 18532 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7a333f8e08c464fa242d908852a172c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 02:11:05.931046 18532 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7a333f8e08c464fa242d908852a172c [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:11:05.931464 18532 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7a333f8e08c464fa242d908852a172c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7a333f8e08c464fa242d908852a172c" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 45119 } }
I20240507 02:11:05.931545 18532 leader_election.cc:302] T 00000000000000000000000000000000 P f7a333f8e08c464fa242d908852a172c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7a333f8e08c464fa242d908852a172c; no voters:
I20240507 02:11:05.931650 18532 leader_election.cc:288] T 00000000000000000000000000000000 P f7a333f8e08c464fa242d908852a172c [CANDIDATE]: Term 1 election: Requested vote from peers
I20240507 02:11:05.931704 18535 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7a333f8e08c464fa242d908852a172c [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:11:05.931840 18535 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7a333f8e08c464fa242d908852a172c [term 1 LEADER]: Becoming Leader. State: Replica: f7a333f8e08c464fa242d908852a172c, State: Running, Role: LEADER
I20240507 02:11:05.931855 18532 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7a333f8e08c464fa242d908852a172c [sys.catalog]: configured and running, proceeding with master startup.
I20240507 02:11:05.931926 18535 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7a333f8e08c464fa242d908852a172c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7a333f8e08c464fa242d908852a172c" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 45119 } }
I20240507 02:11:05.932232 18536 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7a333f8e08c464fa242d908852a172c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7a333f8e08c464fa242d908852a172c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7a333f8e08c464fa242d908852a172c" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 45119 } } }
I20240507 02:11:05.932353 18536 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7a333f8e08c464fa242d908852a172c [sys.catalog]: This master's current role is: LEADER
I20240507 02:11:05.932257 18537 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7a333f8e08c464fa242d908852a172c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7a333f8e08c464fa242d908852a172c. Latest consensus state: current_term: 1 leader_uuid: "f7a333f8e08c464fa242d908852a172c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7a333f8e08c464fa242d908852a172c" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 45119 } } }
I20240507 02:11:05.932482 18537 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7a333f8e08c464fa242d908852a172c [sys.catalog]: This master's current role is: LEADER
I20240507 02:11:05.932698 18544 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 02:11:05.933127 18544 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240507 02:11:05.933501 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 18465
I20240507 02:11:05.933600 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715047604443657-32269-0/minicluster-data/master-0/wal/instance
I20240507 02:11:05.933773 32269 external_mini_cluster.cc:1428] Setting key b13b92b91e91b70b98b65c3d2500596c
I20240507 02:11:05.934919 18544 catalog_manager.cc:1347] Generated new cluster ID: 28c19534abda4473b9c5fea6a72c7142
I20240507 02:11:05.934973 18544 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240507 02:11:05.938148 18554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:05.934155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56898 (local address 127.31.131.126:45119)
0507 02:11:05.934326 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:05.934333 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:05.934351 (+ 18us) server_negotiation.cc:407] Connection header received
0507 02:11:05.934437 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:05.934443 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:05.934494 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:05.934660 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:05.935063 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:05.936199 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:05.937099 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:05.937310 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:05.937536 (+ 226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:05.937551 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:05.937551 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0507 02:11:05.937715 (+ 164us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:05.937738 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0507 02:11:05.937744 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:05.937788 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:05.937954 (+ 166us) server_negotiation.cc:299] Negotiation successful
0507 02:11:05.937998 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":84,"threads_started":1}
I20240507 02:11:05.967000 18544 catalog_manager.cc:1370] Generated new certificate authority record
I20240507 02:11:05.967442 18544 catalog_manager.cc:1504] Loading token signing keys...
I20240507 02:11:05.978170 18544 catalog_manager.cc:5857] T 00000000000000000000000000000000 P f7a333f8e08c464fa242d908852a172c: Generated new TSK 0
I20240507 02:11:05.978307 18544 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240507 02:11:05.985196 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715047604443657-32269-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715047604443657-32269-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715047604443657-32269-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.65:0
--local_ip_for_outbound_sockets=127.31.131.65
--webserver_interface=127.31.131.65
--webserver_port=0
--tserver_master_addrs=127.31.131.126:45119
--builtin_ntp_servers=127.31.131.84:41833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20240507 02:11:06.072484 18556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:11:06.072631 18556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:11:06.072650 18556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:11:06.073855 18556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:11:06.073896 18556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:11:06.073918 18556 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.65
I20240507 02:11:06.075280 18556 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:41833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715047604443657-32269-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715047604443657-32269-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.131.65
--webserver_port=0
--tserver_master_addrs=127.31.131.126:45119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.65
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715047604443657-32269-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:11:06.075443 18556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:11:06.075652 18556 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:11:06.079826 18562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:06.079926 18556 server_base.cc:1034] running on GCE node
W20240507 02:11:06.079815 18561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:11:06.080114 18564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:06.080294 18556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:11:06.080479 18556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:11:06.081617 18556 hybrid_clock.cc:648] HybridClock initialized: now 1715047866081607 us; error 25 us; skew 500 ppm
I20240507 02:11:06.082717 18556 webserver.cc:460] Webserver started at http://127.31.131.65:34897/ using document root <none> and password file <none>
I20240507 02:11:06.082892 18556 fs_manager.cc:359] Metadata directory not provided
I20240507 02:11:06.082926 18556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:11:06.083022 18556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:11:06.083885 18556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715047604443657-32269-0/minicluster-data/ts-0/data/instance:
uuid: "5791895f5e0f45b0bec6d9822d80ed32"
format_stamp: "Formatted at 2024-05-07 02:11:06 on dist-test-slave-v0bg"
server_key: "344297e3bdd9380349ea3650a1c98fc8"
server_key_iv: "1999feb7ef3ca2ede80a08db919273e5"
server_key_version: "encryptionkey@0"
I20240507 02:11:06.084138 18556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715047604443657-32269-0/minicluster-data/ts-0/wal/instance:
uuid: "5791895f5e0f45b0bec6d9822d80ed32"
format_stamp: "Formatted at 2024-05-07 02:11:06 on dist-test-slave-v0bg"
server_key: "344297e3bdd9380349ea3650a1c98fc8"
server_key_iv: "1999feb7ef3ca2ede80a08db919273e5"
server_key_version: "encryptionkey@0"
I20240507 02:11:06.085120 18556 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240507 02:11:06.085700 18570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:06.085855 18556 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240507 02:11:06.085922 18556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715047604443657-32269-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715047604443657-32269-0/minicluster-data/ts-0/wal
uuid: "5791895f5e0f45b0bec6d9822d80ed32"
format_stamp: "Formatted at 2024-05-07 02:11:06 on dist-test-slave-v0bg"
server_key: "344297e3bdd9380349ea3650a1c98fc8"
server_key_iv: "1999feb7ef3ca2ede80a08db919273e5"
server_key_version: "encryptionkey@0"
I20240507 02:11:06.085975 18556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715047604443657-32269-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715047604443657-32269-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715047604443657-32269-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:11:06.102008 18556 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:11:06.102195 18556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:11:06.102283 18556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:11:06.102433 18556 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:11:06.102715 18556 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:11:06.102746 18556 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:06.102770 18556 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:11:06.102794 18556 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:06.107512 18556 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.65:34193
I20240507 02:11:06.107558 18683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.65:34193 every 8 connection(s)
I20240507 02:11:06.107774 18556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
I20240507 02:11:06.109477 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 18556
I20240507 02:11:06.109582 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715047604443657-32269-0/minicluster-data/ts-0/wal/instance
I20240507 02:11:06.109726 32269 external_mini_cluster.cc:1428] Setting key 1e68bdc997f3122963c01c7a8be3a5e2
I20240507 02:11:06.110625 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715047604443657-32269-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715047604443657-32269-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715047604443657-32269-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.66:0
--local_ip_for_outbound_sockets=127.31.131.66
--webserver_interface=127.31.131.66
--webserver_port=0
--tserver_master_addrs=127.31.131.126:45119
--builtin_ntp_servers=127.31.131.84:41833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20240507 02:11:06.111745 18554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:06.108290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:34381 (local address 127.31.131.126:45119)
0507 02:11:06.108345 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:06.108347 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:06.108496 (+ 149us) server_negotiation.cc:407] Connection header received
0507 02:11:06.108873 (+ 377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:06.108874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:06.108883 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:06.108931 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:06.109296 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:06.109798 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:06.110429 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:06.110605 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:06.111526 (+ 921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:06.111530 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:06.111530 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0507 02:11:06.111534 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:06.111547 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0507 02:11:06.111550 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:06.111593 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:06.111668 (+ 75us) server_negotiation.cc:299] Negotiation successful
0507 02:11:06.111692 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20240507 02:11:06.111929 18686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:06.108262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:45119 (local address 127.31.131.65:34381)
0507 02:11:06.108418 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:06.108442 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0507 02:11:06.108802 (+ 360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:11:06.108969 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:11:06.108972 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:11:06.108979 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:11:06.109240 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:06.109243 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:06.109876 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:11:06.109877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:11:06.110357 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:06.110359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:06.110480 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:06.111394 (+ 914us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0507 02:11:06.111395 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:11:06.111415 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0507 02:11:06.111435 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:11:06.111603 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:11:06.111606 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:11:06.111608 (+ 2us) client_negotiation.cc:769] Sending connection context
0507 02:11:06.111670 (+ 62us) client_negotiation.cc:240] Negotiation successful
0507 02:11:06.111729 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":72,"threads_started":1}
I20240507 02:11:06.112352 18684 heartbeater.cc:347] Connected to a master server at 127.31.131.126:45119
I20240507 02:11:06.112437 18684 heartbeater.cc:464] Registering TS with master...
I20240507 02:11:06.112622 18684 heartbeater.cc:511] Master 127.31.131.126:45119 requested a full tablet report, sending...
I20240507 02:11:06.112996 18496 ts_manager.cc:194] Registered new tserver with Master: 5791895f5e0f45b0bec6d9822d80ed32 (127.31.131.65:34193)
I20240507 02:11:06.113586 18496 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.131.65:34381
W20240507 02:11:06.184185 18687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:11:06.184350 18687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:11:06.184370 18687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:11:06.185559 18687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:11:06.185591 18687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:11:06.185612 18687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.66
I20240507 02:11:06.186949 18687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:41833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715047604443657-32269-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715047604443657-32269-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.131.66
--webserver_port=0
--tserver_master_addrs=127.31.131.126:45119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.66
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715047604443657-32269-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:11:06.187108 18687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:11:06.187302 18687 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:11:06.190852 18692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:11:06.190927 18693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:11:06.191037 18695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:06.191150 18687 server_base.cc:1034] running on GCE node
I20240507 02:11:06.191308 18687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:11:06.191504 18687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:11:06.192629 18687 hybrid_clock.cc:648] HybridClock initialized: now 1715047866192618 us; error 23 us; skew 500 ppm
I20240507 02:11:06.193670 18687 webserver.cc:460] Webserver started at http://127.31.131.66:44855/ using document root <none> and password file <none>
I20240507 02:11:06.193828 18687 fs_manager.cc:359] Metadata directory not provided
I20240507 02:11:06.193871 18687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:11:06.193985 18687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:11:06.194923 18687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715047604443657-32269-0/minicluster-data/ts-1/data/instance:
uuid: "4f70a075889c47d489f7b8244336f6e4"
format_stamp: "Formatted at 2024-05-07 02:11:06 on dist-test-slave-v0bg"
server_key: "c7a184f347edfb65b9d16517fc3c5ade"
server_key_iv: "85b47b7ea9e9f935974e4d84e42be9ea"
server_key_version: "encryptionkey@0"
I20240507 02:11:06.195204 18687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715047604443657-32269-0/minicluster-data/ts-1/wal/instance:
uuid: "4f70a075889c47d489f7b8244336f6e4"
format_stamp: "Formatted at 2024-05-07 02:11:06 on dist-test-slave-v0bg"
server_key: "c7a184f347edfb65b9d16517fc3c5ade"
server_key_iv: "85b47b7ea9e9f935974e4d84e42be9ea"
server_key_version: "encryptionkey@0"
I20240507 02:11:06.196233 18687 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240507 02:11:06.196817 18701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:06.196952 18687 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240507 02:11:06.197022 18687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715047604443657-32269-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715047604443657-32269-0/minicluster-data/ts-1/wal
uuid: "4f70a075889c47d489f7b8244336f6e4"
format_stamp: "Formatted at 2024-05-07 02:11:06 on dist-test-slave-v0bg"
server_key: "c7a184f347edfb65b9d16517fc3c5ade"
server_key_iv: "85b47b7ea9e9f935974e4d84e42be9ea"
server_key_version: "encryptionkey@0"
I20240507 02:11:06.197082 18687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715047604443657-32269-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715047604443657-32269-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715047604443657-32269-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:11:06.207582 18687 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:11:06.207755 18687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:11:06.207851 18687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:11:06.208004 18687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:11:06.208257 18687 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:11:06.208290 18687 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:06.208318 18687 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:11:06.208346 18687 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:06.213567 18687 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.66:36093
I20240507 02:11:06.213621 18814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.66:36093 every 8 connection(s)
I20240507 02:11:06.213840 18687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
I20240507 02:11:06.214692 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 18687
I20240507 02:11:06.214787 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715047604443657-32269-0/minicluster-data/ts-1/wal/instance
I20240507 02:11:06.214974 32269 external_mini_cluster.cc:1428] Setting key ed8baed96dc7d14f93fb4f3dd61670f4
I20240507 02:11:06.215867 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715047604443657-32269-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715047604443657-32269-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715047604443657-32269-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.67:0
--local_ip_for_outbound_sockets=127.31.131.67
--webserver_interface=127.31.131.67
--webserver_port=0
--tserver_master_addrs=127.31.131.126:45119
--builtin_ntp_servers=127.31.131.84:41833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20240507 02:11:06.218194 18554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:06.214425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.66:33241 (local address 127.31.131.126:45119)
0507 02:11:06.214470 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:06.214473 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:06.214730 (+ 257us) server_negotiation.cc:407] Connection header received
0507 02:11:06.215247 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:06.215248 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:06.215264 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:06.215320 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:06.215764 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:06.216302 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:06.216958 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:06.217111 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:06.217918 (+ 807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:06.217922 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:06.217922 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0507 02:11:06.217927 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:06.217945 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0507 02:11:06.217949 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:06.218019 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:06.218079 (+ 60us) server_negotiation.cc:299] Negotiation successful
0507 02:11:06.218116 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20240507 02:11:06.218218 18817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:06.214406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:45119 (local address 127.31.131.66:33241)
0507 02:11:06.214622 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:06.214649 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0507 02:11:06.215144 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:11:06.215376 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:11:06.215380 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:11:06.215387 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:11:06.215683 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:06.215687 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:06.216381 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:11:06.216382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:11:06.216886 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:06.216888 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:06.217021 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:06.217803 (+ 782us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0507 02:11:06.217804 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:11:06.217819 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0507 02:11:06.217834 (+ 15us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:11:06.218019 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:11:06.218021 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:11:06.218023 (+ 2us) client_negotiation.cc:769] Sending connection context
0507 02:11:06.218066 (+ 43us) client_negotiation.cc:240] Negotiation successful
0507 02:11:06.218112 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":84,"threads_started":1}
I20240507 02:11:06.218672 18815 heartbeater.cc:347] Connected to a master server at 127.31.131.126:45119
I20240507 02:11:06.218765 18815 heartbeater.cc:464] Registering TS with master...
I20240507 02:11:06.218966 18815 heartbeater.cc:511] Master 127.31.131.126:45119 requested a full tablet report, sending...
I20240507 02:11:06.219324 18496 ts_manager.cc:194] Registered new tserver with Master: 4f70a075889c47d489f7b8244336f6e4 (127.31.131.66:36093)
I20240507 02:11:06.219676 18496 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.131.66:33241
W20240507 02:11:06.290930 18818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:11:06.291065 18818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:11:06.291083 18818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:11:06.292296 18818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:11:06.292331 18818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:11:06.292354 18818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.67
I20240507 02:11:06.293727 18818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:41833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715047604443657-32269-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715047604443657-32269-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.131.67
--webserver_port=0
--tserver_master_addrs=127.31.131.126:45119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.67
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715047604443657-32269-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:11:06.293892 18818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:11:06.294118 18818 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:11:06.297717 18826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:11:06.297706 18824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:11:06.297832 18823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:06.297897 18818 server_base.cc:1034] running on GCE node
I20240507 02:11:06.298053 18818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:11:06.298225 18818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:11:06.299350 18818 hybrid_clock.cc:648] HybridClock initialized: now 1715047866299339 us; error 22 us; skew 500 ppm
I20240507 02:11:06.300390 18818 webserver.cc:460] Webserver started at http://127.31.131.67:44983/ using document root <none> and password file <none>
I20240507 02:11:06.300549 18818 fs_manager.cc:359] Metadata directory not provided
I20240507 02:11:06.300594 18818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:11:06.300689 18818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:11:06.301538 18818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715047604443657-32269-0/minicluster-data/ts-2/data/instance:
uuid: "c776858b541f4435b0b2447ebc8da417"
format_stamp: "Formatted at 2024-05-07 02:11:06 on dist-test-slave-v0bg"
server_key: "29c01318fc5f3d32e7acd32406fd8cad"
server_key_iv: "b2bbe62c4a21fe3776f360e530e01394"
server_key_version: "encryptionkey@0"
I20240507 02:11:06.301800 18818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715047604443657-32269-0/minicluster-data/ts-2/wal/instance:
uuid: "c776858b541f4435b0b2447ebc8da417"
format_stamp: "Formatted at 2024-05-07 02:11:06 on dist-test-slave-v0bg"
server_key: "29c01318fc5f3d32e7acd32406fd8cad"
server_key_iv: "b2bbe62c4a21fe3776f360e530e01394"
server_key_version: "encryptionkey@0"
I20240507 02:11:06.302781 18818 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240507 02:11:06.303320 18832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:06.303478 18818 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240507 02:11:06.303553 18818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715047604443657-32269-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715047604443657-32269-0/minicluster-data/ts-2/wal
uuid: "c776858b541f4435b0b2447ebc8da417"
format_stamp: "Formatted at 2024-05-07 02:11:06 on dist-test-slave-v0bg"
server_key: "29c01318fc5f3d32e7acd32406fd8cad"
server_key_iv: "b2bbe62c4a21fe3776f360e530e01394"
server_key_version: "encryptionkey@0"
I20240507 02:11:06.303598 18818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715047604443657-32269-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715047604443657-32269-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715047604443657-32269-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:11:06.320268 18818 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:11:06.320434 18818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:11:06.320518 18818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:11:06.320655 18818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:11:06.320868 18818 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:11:06.320892 18818 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:06.320919 18818 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:11:06.320932 18818 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:06.325392 18818 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.67:33061
I20240507 02:11:06.325444 18945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.67:33061 every 8 connection(s)
I20240507 02:11:06.325661 18818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
I20240507 02:11:06.329217 18554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:06.326180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.67:41233 (local address 127.31.131.126:45119)
0507 02:11:06.326226 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:06.326228 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:06.326417 (+ 189us) server_negotiation.cc:407] Connection header received
0507 02:11:06.326820 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:06.326821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:06.326829 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:06.326880 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:06.327265 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:06.327726 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:06.328170 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:06.328265 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:06.329031 (+ 766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:06.329035 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:06.329035 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0507 02:11:06.329039 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:06.329048 (+ 9us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0507 02:11:06.329051 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:06.329081 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:06.329144 (+ 63us) server_negotiation.cc:299] Negotiation successful
0507 02:11:06.329164 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20240507 02:11:06.329304 18948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:06.326159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:45119 (local address 127.31.131.67:41233)
0507 02:11:06.326333 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:06.326356 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0507 02:11:06.326745 (+ 389us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:11:06.326928 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:11:06.326931 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:11:06.326939 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:11:06.327212 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:06.327214 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:06.327778 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:11:06.327778 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:11:06.328123 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:06.328124 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:06.328208 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:06.328935 (+ 727us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0507 02:11:06.328935 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:11:06.328950 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0507 02:11:06.328966 (+ 16us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:11:06.329092 (+ 126us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:11:06.329094 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:11:06.329096 (+ 2us) client_negotiation.cc:769] Sending connection context
0507 02:11:06.329144 (+ 48us) client_negotiation.cc:240] Negotiation successful
0507 02:11:06.329188 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":65,"threads_started":1}
I20240507 02:11:06.329569 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 18818
I20240507 02:11:06.329661 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715047604443657-32269-0/minicluster-data/ts-2/wal/instance
I20240507 02:11:06.329749 18946 heartbeater.cc:347] Connected to a master server at 127.31.131.126:45119
I20240507 02:11:06.329823 32269 external_mini_cluster.cc:1428] Setting key 03ea3932d6751718cd86f90e2cd7a687
I20240507 02:11:06.329833 18946 heartbeater.cc:464] Registering TS with master...
I20240507 02:11:06.329998 18946 heartbeater.cc:511] Master 127.31.131.126:45119 requested a full tablet report, sending...
I20240507 02:11:06.330314 18496 ts_manager.cc:194] Registered new tserver with Master: c776858b541f4435b0b2447ebc8da417 (127.31.131.67:33061)
I20240507 02:11:06.330684 32269 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240507 02:11:06.330775 18496 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.131.67:41233
I20240507 02:11:06.333997 18554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:06.331887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56900 (local address 127.31.131.126:45119)
0507 02:11:06.331925 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:06.331926 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:06.332012 (+ 86us) server_negotiation.cc:407] Connection header received
0507 02:11:06.332053 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:06.332054 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:06.332060 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:06.332101 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:06.332343 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:06.332868 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:06.333364 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:06.333454 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:06.333834 (+ 380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:06.333836 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:06.333837 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0507 02:11:06.333840 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:06.333848 (+ 8us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0507 02:11:06.333851 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:06.333893 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:06.333942 (+ 49us) server_negotiation.cc:299] Negotiation successful
0507 02:11:06.333950 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20240507 02:11:06.335340 18496 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56900:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 02:11:06.336620 18496 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 02:11:06.340811 18954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:06.338212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.67:33061 (local address 127.0.0.1:47168)
0507 02:11:06.338306 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:06.338328 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0507 02:11:06.338385 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:11:06.338757 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:11:06.338759 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:11:06.338761 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:11:06.338972 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:06.338973 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:06.339896 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:11:06.339897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:11:06.340576 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:06.340578 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:06.340647 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:06.340670 (+ 23us) client_negotiation.cc:769] Sending connection context
0507 02:11:06.340712 (+ 42us) client_negotiation.cc:240] Negotiation successful
0507 02:11:06.340747 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73,"thread_start_us":52,"threads_started":1}
I20240507 02:11:06.341228 18956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:06.338232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.66:36093 (local address 127.0.0.1:57794)
0507 02:11:06.338597 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:06.338607 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0507 02:11:06.338664 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:11:06.338814 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:11:06.338814 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:11:06.338816 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:11:06.338991 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:06.338993 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:06.340267 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:11:06.340268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:11:06.341051 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:06.341053 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:06.341108 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:06.341111 (+ 3us) client_negotiation.cc:769] Sending connection context
0507 02:11:06.341137 (+ 26us) client_negotiation.cc:240] Negotiation successful
0507 02:11:06.341171 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":77,"threads_started":1}
I20240507 02:11:06.341356 18955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:06.338233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.65:34193 (local address 127.0.0.1:56374)
0507 02:11:06.338736 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:06.338743 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0507 02:11:06.338785 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:11:06.338959 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:11:06.338960 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:11:06.338962 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:11:06.339149 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:06.339151 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:06.340391 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:11:06.340392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:11:06.341160 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:06.341163 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:06.341226 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:06.341229 (+ 3us) client_negotiation.cc:769] Sending connection context
0507 02:11:06.341256 (+ 27us) client_negotiation.cc:240] Negotiation successful
0507 02:11:06.341292 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":90,"threads_started":1}
I20240507 02:11:06.341594 18958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:06.338450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47168 (local address 127.31.131.67:33061)
0507 02:11:06.338604 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:06.338611 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:06.338621 (+ 10us) server_negotiation.cc:407] Connection header received
0507 02:11:06.338649 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:06.338651 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:06.338664 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:06.338737 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:11:06.339052 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:06.339824 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:06.340654 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:06.341393 (+ 739us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:06.341469 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:06.341497 (+ 28us) server_negotiation.cc:299] Negotiation successful
0507 02:11:06.341530 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":73,"threads_started":1}
I20240507 02:11:06.341998 18957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:06.338425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57794 (local address 127.31.131.66:36093)
0507 02:11:06.338620 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:06.338628 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:06.338652 (+ 24us) server_negotiation.cc:407] Connection header received
0507 02:11:06.338721 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:06.338723 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:06.338741 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:06.338817 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:11:06.339077 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:06.340190 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:06.341370 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:06.341845 (+ 475us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:06.341898 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:06.341922 (+ 24us) server_negotiation.cc:299] Negotiation successful
0507 02:11:06.341947 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":79,"threads_started":1}
I20240507 02:11:06.342048 18880 tablet_service.cc:1467] Processing CreateTablet for tablet fdcad1a6e7bf4d9389712f364d4f3958 (DEFAULT_TABLE table=test-table [id=ecdce788540343959517e508766c7df1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:11:06.342298 18880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdcad1a6e7bf4d9389712f364d4f3958. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:11:06.342314 18959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:06.338458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56374 (local address 127.31.131.65:34193)
0507 02:11:06.338634 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:06.338640 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:06.338835 (+ 195us) server_negotiation.cc:407] Connection header received
0507 02:11:06.338873 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:06.338874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:06.338889 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:06.338948 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:11:06.339233 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:06.340321 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:06.341502 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:06.342117 (+ 615us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:06.342185 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:06.342214 (+ 29us) server_negotiation.cc:299] Negotiation successful
0507 02:11:06.342248 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":103,"threads_started":1}
I20240507 02:11:06.342429 18749 tablet_service.cc:1467] Processing CreateTablet for tablet fdcad1a6e7bf4d9389712f364d4f3958 (DEFAULT_TABLE table=test-table [id=ecdce788540343959517e508766c7df1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:11:06.342700 18749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdcad1a6e7bf4d9389712f364d4f3958. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:11:06.342797 18618 tablet_service.cc:1467] Processing CreateTablet for tablet fdcad1a6e7bf4d9389712f364d4f3958 (DEFAULT_TABLE table=test-table [id=ecdce788540343959517e508766c7df1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:11:06.343027 18618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdcad1a6e7bf4d9389712f364d4f3958. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:11:06.344561 18960 tablet_bootstrap.cc:492] T fdcad1a6e7bf4d9389712f364d4f3958 P c776858b541f4435b0b2447ebc8da417: Bootstrap starting.
I20240507 02:11:06.345001 18960 tablet_bootstrap.cc:654] T fdcad1a6e7bf4d9389712f364d4f3958 P c776858b541f4435b0b2447ebc8da417: Neither blocks nor log segments found. Creating new log.
I20240507 02:11:06.345214 18960 log.cc:824] T fdcad1a6e7bf4d9389712f364d4f3958 P c776858b541f4435b0b2447ebc8da417: Log is configured to *not* fsync() on all Append() calls
I20240507 02:11:06.345458 18961 tablet_bootstrap.cc:492] T fdcad1a6e7bf4d9389712f364d4f3958 P 4f70a075889c47d489f7b8244336f6e4: Bootstrap starting.
I20240507 02:11:06.345844 18963 tablet_bootstrap.cc:492] T fdcad1a6e7bf4d9389712f364d4f3958 P 5791895f5e0f45b0bec6d9822d80ed32: Bootstrap starting.
I20240507 02:11:06.345925 18960 tablet_bootstrap.cc:492] T fdcad1a6e7bf4d9389712f364d4f3958 P c776858b541f4435b0b2447ebc8da417: No bootstrap required, opened a new log
I20240507 02:11:06.345986 18960 ts_tablet_manager.cc:1367] T fdcad1a6e7bf4d9389712f364d4f3958 P c776858b541f4435b0b2447ebc8da417: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240507 02:11:06.346071 18961 tablet_bootstrap.cc:654] T fdcad1a6e7bf4d9389712f364d4f3958 P 4f70a075889c47d489f7b8244336f6e4: Neither blocks nor log segments found. Creating new log.
I20240507 02:11:06.346290 18963 tablet_bootstrap.cc:654] T fdcad1a6e7bf4d9389712f364d4f3958 P 5791895f5e0f45b0bec6d9822d80ed32: Neither blocks nor log segments found. Creating new log.
I20240507 02:11:06.346328 18961 log.cc:824] T fdcad1a6e7bf4d9389712f364d4f3958 P 4f70a075889c47d489f7b8244336f6e4: Log is configured to *not* fsync() on all Append() calls
I20240507 02:11:06.346530 18963 log.cc:824] T fdcad1a6e7bf4d9389712f364d4f3958 P 5791895f5e0f45b0bec6d9822d80ed32: Log is configured to *not* fsync() on all Append() calls
I20240507 02:11:06.347074 18961 tablet_bootstrap.cc:492] T fdcad1a6e7bf4d9389712f364d4f3958 P 4f70a075889c47d489f7b8244336f6e4: No bootstrap required, opened a new log
I20240507 02:11:06.347082 18963 tablet_bootstrap.cc:492] T fdcad1a6e7bf4d9389712f364d4f3958 P 5791895f5e0f45b0bec6d9822d80ed32: No bootstrap required, opened a new log
I20240507 02:11:06.347136 18963 ts_tablet_manager.cc:1367] T fdcad1a6e7bf4d9389712f364d4f3958 P 5791895f5e0f45b0bec6d9822d80ed32: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240507 02:11:06.347136 18961 ts_tablet_manager.cc:1367] T fdcad1a6e7bf4d9389712f364d4f3958 P 4f70a075889c47d489f7b8244336f6e4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240507 02:11:06.348084 18960 raft_consensus.cc:348] T fdcad1a6e7bf4d9389712f364d4f3958 P c776858b541f4435b0b2447ebc8da417 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c776858b541f4435b0b2447ebc8da417" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 33061 } } peers { permanent_uuid: "5791895f5e0f45b0bec6d9822d80ed32" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 34193 } } peers { permanent_uuid: "4f70a075889c47d489f7b8244336f6e4" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 36093 } }
I20240507 02:11:06.348215 18960 raft_consensus.cc:374] T fdcad1a6e7bf4d9389712f364d4f3958 P c776858b541f4435b0b2447ebc8da417 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:11:06.348238 18960 raft_consensus.cc:729] T fdcad1a6e7bf4d9389712f364d4f3958 P c776858b541f4435b0b2447ebc8da417 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c776858b541f4435b0b2447ebc8da417, State: Initialized, Role: FOLLOWER
I20240507 02:11:06.348317 18960 consensus_queue.cc:260] T fdcad1a6e7bf4d9389712f364d4f3958 P c776858b541f4435b0b2447ebc8da417 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c776858b541f4435b0b2447ebc8da417" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 33061 } } peers { permanent_uuid: "5791895f5e0f45b0bec6d9822d80ed32" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 34193 } } peers { permanent_uuid: "4f70a075889c47d489f7b8244336f6e4" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 36093 } }
I20240507 02:11:06.348553 18960 ts_tablet_manager.cc:1398] T fdcad1a6e7bf4d9389712f364d4f3958 P c776858b541f4435b0b2447ebc8da417: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240507 02:11:06.348594 18946 heartbeater.cc:503] Master 127.31.131.126:45119 was elected leader, sending a full tablet report...
I20240507 02:11:06.348716 18963 raft_consensus.cc:348] T fdcad1a6e7bf4d9389712f364d4f3958 P 5791895f5e0f45b0bec6d9822d80ed32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c776858b541f4435b0b2447ebc8da417" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 33061 } } peers { permanent_uuid: "5791895f5e0f45b0bec6d9822d80ed32" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 34193 } } peers { permanent_uuid: "4f70a075889c47d489f7b8244336f6e4" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 36093 } }
I20240507 02:11:06.348825 18963 raft_consensus.cc:374] T fdcad1a6e7bf4d9389712f364d4f3958 P 5791895f5e0f45b0bec6d9822d80ed32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:11:06.348850 18963 raft_consensus.cc:729] T fdcad1a6e7bf4d9389712f364d4f3958 P 5791895f5e0f45b0bec6d9822d80ed32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5791895f5e0f45b0bec6d9822d80ed32, State: Initialized, Role: FOLLOWER
I20240507 02:11:06.348923 18963 consensus_queue.cc:260] T fdcad1a6e7bf4d9389712f364d4f3958 P 5791895f5e0f45b0bec6d9822d80ed32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c776858b541f4435b0b2447ebc8da417" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 33061 } } peers { permanent_uuid: "5791895f5e0f45b0bec6d9822d80ed32" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 34193 } } peers { permanent_uuid: "4f70a075889c47d489f7b8244336f6e4" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 36093 } }
I20240507 02:11:06.349126 18963 ts_tablet_manager.cc:1398] T fdcad1a6e7bf4d9389712f364d4f3958 P 5791895f5e0f45b0bec6d9822d80ed32: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240507 02:11:06.349160 18961 raft_consensus.cc:348] T fdcad1a6e7bf4d9389712f364d4f3958 P 4f70a075889c47d489f7b8244336f6e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c776858b541f4435b0b2447ebc8da417" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 33061 } } peers { permanent_uuid: "5791895f5e0f45b0bec6d9822d80ed32" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 34193 } } peers { permanent_uuid: "4f70a075889c47d489f7b8244336f6e4" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 36093 } }
I20240507 02:11:06.349248 18684 heartbeater.cc:503] Master 127.31.131.126:45119 was elected leader, sending a full tablet report...
I20240507 02:11:06.349279 18961 raft_consensus.cc:374] T fdcad1a6e7bf4d9389712f364d4f3958 P 4f70a075889c47d489f7b8244336f6e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:11:06.349305 18961 raft_consensus.cc:729] T fdcad1a6e7bf4d9389712f364d4f3958 P 4f70a075889c47d489f7b8244336f6e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f70a075889c47d489f7b8244336f6e4, State: Initialized, Role: FOLLOWER
I20240507 02:11:06.349396 18961 consensus_queue.cc:260] T fdcad1a6e7bf4d9389712f364d4f3958 P 4f70a075889c47d489f7b8244336f6e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c776858b541f4435b0b2447ebc8da417" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 33061 } } peers { permanent_uuid: "5791895f5e0f45b0bec6d9822d80ed32" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 34193 } } peers { permanent_uuid: "4f70a075889c47d489f7b8244336f6e4" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 36093 } }
I20240507 02:11:06.349596 18961 ts_tablet_manager.cc:1398] T fdcad1a6e7bf4d9389712f364d4f3958 P 4f70a075889c47d489f7b8244336f6e4: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240507 02:11:06.349772 18815 heartbeater.cc:503] Master 127.31.131.126:45119 was elected leader, sending a full tablet report...
W20240507 02:11:06.358161 18685 tablet.cc:2369] T fdcad1a6e7bf4d9389712f364d4f3958 P 5791895f5e0f45b0bec6d9822d80ed32: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240507 02:11:06.453161 18966 raft_consensus.cc:482] T fdcad1a6e7bf4d9389712f364d4f3958 P c776858b541f4435b0b2447ebc8da417 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 02:11:06.453253 18966 raft_consensus.cc:504] T fdcad1a6e7bf4d9389712f364d4f3958 P c776858b541f4435b0b2447ebc8da417 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c776858b541f4435b0b2447ebc8da417" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 33061 } } peers { permanent_uuid: "5791895f5e0f45b0bec6d9822d80ed32" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 34193 } } peers { permanent_uuid: "4f70a075889c47d489f7b8244336f6e4" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 36093 } }
I20240507 02:11:06.453519 18966 leader_election.cc:288] T fdcad1a6e7bf4d9389712f364d4f3958 P c776858b541f4435b0b2447ebc8da417 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5791895f5e0f45b0bec6d9822d80ed32 (127.31.131.65:34193), 4f70a075889c47d489f7b8244336f6e4 (127.31.131.66:36093)
I20240507 02:11:06.455574 18948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:06.453647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.65:34193 (local address 127.31.131.67:55921)
0507 02:11:06.453703 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:06.453712 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0507 02:11:06.453749 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:11:06.453947 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:11:06.453948 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:11:06.453950 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:11:06.454120 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:06.454122 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:06.454854 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:11:06.454855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:11:06.455385 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:06.455387 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:06.455444 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:06.455448 (+ 4us) client_negotiation.cc:769] Sending connection context
0507 02:11:06.455474 (+ 26us) client_negotiation.cc:240] Negotiation successful
0507 02:11:06.455508 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20240507 02:11:06.455763 18969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:06.453667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.66:36093 (local address 127.31.131.67:33745)
0507 02:11:06.453795 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:06.453804 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0507 02:11:06.453857 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:11:06.453981 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:11:06.453981 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:11:06.453983 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:11:06.454160 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:06.454161 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:06.454840 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:11:06.454841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:11:06.455397 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:06.455398 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:06.455474 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:06.455477 (+ 3us) client_negotiation.cc:769] Sending connection context
0507 02:11:06.455505 (+ 28us) client_negotiation.cc:240] Negotiation successful
0507 02:11:06.455555 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":72,"threads_started":1}
I20240507 02:11:06.455986 18959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:06.453713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.67:55921 (local address 127.31.131.65:34193)
0507 02:11:06.453752 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:06.453754 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:06.453762 (+ 8us) server_negotiation.cc:407] Connection header received
0507 02:11:06.453793 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:06.453794 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:06.453804 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:06.453867 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:11:06.454195 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:06.454759 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:06.455465 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:06.455894 (+ 429us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:06.455902 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:06.455922 (+ 20us) server_negotiation.cc:299] Negotiation successful
0507 02:11:06.455943 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20240507 02:11:06.456012 18957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:06.453721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.67:33745 (local address 127.31.131.66:36093)
0507 02:11:06.453755 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:06.453756 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:06.453870 (+ 114us) server_negotiation.cc:407] Connection header received
0507 02:11:06.453897 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:06.453898 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:06.453910 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:06.453968 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:11:06.454253 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:06.454751 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:06.455476 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:06.455911 (+ 435us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:06.455919 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:06.455940 (+ 21us) server_negotiation.cc:299] Negotiation successful
0507 02:11:06.455969 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20240507 02:11:06.456254 18638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdcad1a6e7bf4d9389712f364d4f3958" candidate_uuid: "c776858b541f4435b0b2447ebc8da417" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5791895f5e0f45b0bec6d9822d80ed32" is_pre_election: true
I20240507 02:11:06.456260 18769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdcad1a6e7bf4d9389712f364d4f3958" candidate_uuid: "c776858b541f4435b0b2447ebc8da417" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f70a075889c47d489f7b8244336f6e4" is_pre_election: true
I20240507 02:11:06.456355 18769 raft_consensus.cc:2454] T fdcad1a6e7bf4d9389712f364d4f3958 P 4f70a075889c47d489f7b8244336f6e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c776858b541f4435b0b2447ebc8da417 in term 0.
I20240507 02:11:06.456367 18638 raft_consensus.cc:2454] T fdcad1a6e7bf4d9389712f364d4f3958 P 5791895f5e0f45b0bec6d9822d80ed32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c776858b541f4435b0b2447ebc8da417 in term 0.
I20240507 02:11:06.456552 18835 leader_election.cc:302] T fdcad1a6e7bf4d9389712f364d4f3958 P c776858b541f4435b0b2447ebc8da417 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f70a075889c47d489f7b8244336f6e4, c776858b541f4435b0b2447ebc8da417; no voters:
I20240507 02:11:06.456694 18966 raft_consensus.cc:2781] T fdcad1a6e7bf4d9389712f364d4f3958 P c776858b541f4435b0b2447ebc8da417 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 02:11:06.456750 18966 raft_consensus.cc:482] T fdcad1a6e7bf4d9389712f364d4f3958 P c776858b541f4435b0b2447ebc8da417 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 02:11:06.456765 18966 raft_consensus.cc:3037] T fdcad1a6e7bf4d9389712f364d4f3958 P c776858b541f4435b0b2447ebc8da417 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:11:06.457361 18966 raft_consensus.cc:504] T fdcad1a6e7bf4d9389712f364d4f3958 P c776858b541f4435b0b2447ebc8da417 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c776858b541f4435b0b2447ebc8da417" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 33061 } } peers { permanent_uuid: "5791895f5e0f45b0bec6d9822d80ed32" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 34193 } } peers { permanent_uuid: "4f70a075889c47d489f7b8244336f6e4" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 36093 } }
I20240507 02:11:06.457471 18966 leader_election.cc:288] T fdcad1a6e7bf4d9389712f364d4f3958 P c776858b541f4435b0b2447ebc8da417 [CANDIDATE]: Term 1 election: Requested vote from peers 5791895f5e0f45b0bec6d9822d80ed32 (127.31.131.65:34193), 4f70a075889c47d489f7b8244336f6e4 (127.31.131.66:36093)
I20240507 02:11:06.457602 18769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdcad1a6e7bf4d9389712f364d4f3958" candidate_uuid: "c776858b541f4435b0b2447ebc8da417" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f70a075889c47d489f7b8244336f6e4"
I20240507 02:11:06.457597 18638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdcad1a6e7bf4d9389712f364d4f3958" candidate_uuid: "c776858b541f4435b0b2447ebc8da417" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5791895f5e0f45b0bec6d9822d80ed32"
I20240507 02:11:06.457662 18638 raft_consensus.cc:3037] T fdcad1a6e7bf4d9389712f364d4f3958 P 5791895f5e0f45b0bec6d9822d80ed32 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:11:06.457664 18769 raft_consensus.cc:3037] T fdcad1a6e7bf4d9389712f364d4f3958 P 4f70a075889c47d489f7b8244336f6e4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:11:06.458257 18769 raft_consensus.cc:2454] T fdcad1a6e7bf4d9389712f364d4f3958 P 4f70a075889c47d489f7b8244336f6e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c776858b541f4435b0b2447ebc8da417 in term 1.
I20240507 02:11:06.458267 18638 raft_consensus.cc:2454] T fdcad1a6e7bf4d9389712f364d4f3958 P 5791895f5e0f45b0bec6d9822d80ed32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c776858b541f4435b0b2447ebc8da417 in term 1.
I20240507 02:11:06.458413 18835 leader_election.cc:302] T fdcad1a6e7bf4d9389712f364d4f3958 P c776858b541f4435b0b2447ebc8da417 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f70a075889c47d489f7b8244336f6e4, c776858b541f4435b0b2447ebc8da417; no voters:
I20240507 02:11:06.458501 18966 raft_consensus.cc:2781] T fdcad1a6e7bf4d9389712f364d4f3958 P c776858b541f4435b0b2447ebc8da417 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:11:06.458638 18966 raft_consensus.cc:686] T fdcad1a6e7bf4d9389712f364d4f3958 P c776858b541f4435b0b2447ebc8da417 [term 1 LEADER]: Becoming Leader. State: Replica: c776858b541f4435b0b2447ebc8da417, State: Running, Role: LEADER
I20240507 02:11:06.458709 18966 consensus_queue.cc:237] T fdcad1a6e7bf4d9389712f364d4f3958 P c776858b541f4435b0b2447ebc8da417 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c776858b541f4435b0b2447ebc8da417" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 33061 } } peers { permanent_uuid: "5791895f5e0f45b0bec6d9822d80ed32" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 34193 } } peers { permanent_uuid: "4f70a075889c47d489f7b8244336f6e4" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 36093 } }
I20240507 02:11:06.459383 18495 catalog_manager.cc:5484] T fdcad1a6e7bf4d9389712f364d4f3958 P c776858b541f4435b0b2447ebc8da417 reported cstate change: term changed from 0 to 1, leader changed from <none> to c776858b541f4435b0b2447ebc8da417 (127.31.131.67). New cstate: current_term: 1 leader_uuid: "c776858b541f4435b0b2447ebc8da417" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c776858b541f4435b0b2447ebc8da417" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 33061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5791895f5e0f45b0bec6d9822d80ed32" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f70a075889c47d489f7b8244336f6e4" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 36093 } health_report { overall_health: UNKNOWN } } }
W20240507 02:11:06.464262 18816 tablet.cc:2369] T fdcad1a6e7bf4d9389712f364d4f3958 P 4f70a075889c47d489f7b8244336f6e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240507 02:11:06.486531 18958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:06.484711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47182 (local address 127.31.131.67:33061)
0507 02:11:06.484858 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:06.484860 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:06.484868 (+ 8us) server_negotiation.cc:407] Connection header received
0507 02:11:06.484894 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:06.484894 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:06.484909 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:06.484973 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0507 02:11:06.485163 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:06.485614 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:06.486104 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:06.486207 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:06.486319 (+ 112us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0507 02:11:06.486388 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0507 02:11:06.486444 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:06.486480 (+ 36us) server_negotiation.cc:299] Negotiation successful
0507 02:11:06.486488 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20240507 02:11:06.487908 18769 raft_consensus.cc:1261] T fdcad1a6e7bf4d9389712f364d4f3958 P 4f70a075889c47d489f7b8244336f6e4 [term 1 FOLLOWER]: Refusing update from remote peer c776858b541f4435b0b2447ebc8da417: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 02:11:06.487933 18638 raft_consensus.cc:1261] T fdcad1a6e7bf4d9389712f364d4f3958 P 5791895f5e0f45b0bec6d9822d80ed32 [term 1 FOLLOWER]: Refusing update from remote peer c776858b541f4435b0b2447ebc8da417: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 02:11:06.488227 18970 consensus_queue.cc:1035] T fdcad1a6e7bf4d9389712f364d4f3958 P c776858b541f4435b0b2447ebc8da417 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5791895f5e0f45b0bec6d9822d80ed32" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 02:11:06.488335 18966 consensus_queue.cc:1035] T fdcad1a6e7bf4d9389712f364d4f3958 P c776858b541f4435b0b2447ebc8da417 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f70a075889c47d489f7b8244336f6e4" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 36093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 02:11:06.490258 18973 mvcc.cc:205] Tried to move back new op lower bound from 7024836061132472320 to 7024836061015416832. Current Snapshot: MvccSnapshot[applied={T|T < 7024836061132472320}]
I20240507 02:11:06.490320 18975 mvcc.cc:205] Tried to move back new op lower bound from 7024836061132472320 to 7024836061015416832. Current Snapshot: MvccSnapshot[applied={T|T < 7024836061132472320}]
I20240507 02:11:06.490404 18974 mvcc.cc:205] Tried to move back new op lower bound from 7024836061132472320 to 7024836061015416832. Current Snapshot: MvccSnapshot[applied={T|T < 7024836061132472320}]
I20240507 02:11:06.492677 18496 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56900:
table { table_name: "test-table" } modify_external_catalogs: true
I20240507 02:11:06.492748 18496 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56900:
table { table_name: "test-table" } modify_external_catalogs: true
I20240507 02:11:06.493359 18496 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f7a333f8e08c464fa242d908852a172c: Sending DeleteTablet for 3 replicas of tablet fdcad1a6e7bf4d9389712f364d4f3958
I20240507 02:11:06.493667 18749 tablet_service.cc:1514] Processing DeleteTablet for tablet fdcad1a6e7bf4d9389712f364d4f3958 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:11:06 UTC) from {username='slave'} at 127.0.0.1:57794
I20240507 02:11:06.493719 18618 tablet_service.cc:1514] Processing DeleteTablet for tablet fdcad1a6e7bf4d9389712f364d4f3958 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:11:06 UTC) from {username='slave'} at 127.0.0.1:56374
I20240507 02:11:06.493867 18983 tablet_replica.cc:330] T fdcad1a6e7bf4d9389712f364d4f3958 P 4f70a075889c47d489f7b8244336f6e4: stopping tablet replica
I20240507 02:11:06.493940 18983 raft_consensus.cc:2229] T fdcad1a6e7bf4d9389712f364d4f3958 P 4f70a075889c47d489f7b8244336f6e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 02:11:06.493955 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 18556
I20240507 02:11:06.494009 18983 raft_consensus.cc:2258] T fdcad1a6e7bf4d9389712f364d4f3958 P 4f70a075889c47d489f7b8244336f6e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 02:11:06.495560 18983 ts_tablet_manager.cc:1875] T fdcad1a6e7bf4d9389712f364d4f3958 P 4f70a075889c47d489f7b8244336f6e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 02:11:06.495909 18880 tablet_service.cc:1514] Processing DeleteTablet for tablet fdcad1a6e7bf4d9389712f364d4f3958 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:11:06 UTC) from {username='slave'} at 127.0.0.1:47168
I20240507 02:11:06.496084 18985 tablet_replica.cc:330] T fdcad1a6e7bf4d9389712f364d4f3958 P c776858b541f4435b0b2447ebc8da417: stopping tablet replica
I20240507 02:11:06.496148 18985 raft_consensus.cc:2229] T fdcad1a6e7bf4d9389712f364d4f3958 P c776858b541f4435b0b2447ebc8da417 [term 1 LEADER]: Raft consensus shutting down.
I20240507 02:11:06.496233 18985 raft_consensus.cc:2258] T fdcad1a6e7bf4d9389712f364d4f3958 P c776858b541f4435b0b2447ebc8da417 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 02:11:06.496593 18985 ts_tablet_manager.cc:1875] T fdcad1a6e7bf4d9389712f364d4f3958 P c776858b541f4435b0b2447ebc8da417: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 02:11:06.496852 18983 ts_tablet_manager.cc:1888] T fdcad1a6e7bf4d9389712f364d4f3958 P 4f70a075889c47d489f7b8244336f6e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240507 02:11:06.496956 18983 log.cc:1196] T fdcad1a6e7bf4d9389712f364d4f3958 P 4f70a075889c47d489f7b8244336f6e4: Deleting WAL directory at /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715047604443657-32269-0/minicluster-data/ts-1/wal/wals/fdcad1a6e7bf4d9389712f364d4f3958
I20240507 02:11:06.497280 18983 ts_tablet_manager.cc:1909] T fdcad1a6e7bf4d9389712f364d4f3958 P 4f70a075889c47d489f7b8244336f6e4: Deleting consensus metadata
I20240507 02:11:06.497649 18482 catalog_manager.cc:4830] TS 4f70a075889c47d489f7b8244336f6e4 (127.31.131.66:36093): tablet fdcad1a6e7bf4d9389712f364d4f3958 (table test-table [id=ecdce788540343959517e508766c7df1]) successfully deleted
I20240507 02:11:06.498042 18985 ts_tablet_manager.cc:1888] T fdcad1a6e7bf4d9389712f364d4f3958 P c776858b541f4435b0b2447ebc8da417: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240507 02:11:06.498102 18985 log.cc:1196] T fdcad1a6e7bf4d9389712f364d4f3958 P c776858b541f4435b0b2447ebc8da417: Deleting WAL directory at /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1715047604443657-32269-0/minicluster-data/ts-2/wal/wals/fdcad1a6e7bf4d9389712f364d4f3958
I20240507 02:11:06.498314 18985 ts_tablet_manager.cc:1909] T fdcad1a6e7bf4d9389712f364d4f3958 P c776858b541f4435b0b2447ebc8da417: Deleting consensus metadata
I20240507 02:11:06.498625 18483 catalog_manager.cc:4830] TS c776858b541f4435b0b2447ebc8da417 (127.31.131.67:33061): tablet fdcad1a6e7bf4d9389712f364d4f3958 (table test-table [id=ecdce788540343959517e508766c7df1]) successfully deleted
W20240507 02:11:06.499532 18481 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.131.65:34193 (error 108)
I20240507 02:11:06.499936 18955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:06.499830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.65:34193 (local address 127.0.0.1:56376)
0507 02:11:06.499876 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:06.499907 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.65:34193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20240507 02:11:06.499938 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 18687
W20240507 02:11:06.501163 18481 catalog_manager.cc:4573] TS 5791895f5e0f45b0bec6d9822d80ed32 (127.31.131.65:34193): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fdcad1a6e7bf4d9389712f364d4f3958: Network error: Client connection negotiation failed: client connection to 127.31.131.65:34193: connect: Connection refused (error 111)
I20240507 02:11:06.504737 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 18818
I20240507 02:11:06.508342 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 18465
2024-05-07T02:11:06Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (710 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 07 02:11:06 dist-test-slave-v0bg krb5kdc[18989](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 07 02:11:06 dist-test-slave-v0bg krb5kdc[18989](info): set up 2 sockets
May 07 02:11:06 dist-test-slave-v0bg krb5kdc[18989](info): commencing operation
krb5kdc: starting...
W20240507 02:11:07.386945 32269 mini_kdc.cc:121] Time spent starting KDC: real 0.861s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/krb5kdc/test-user.keytab.
May 07 02:11:07 dist-test-slave-v0bg krb5kdc[18989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047867, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-07T02:11:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-07T02:11:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.131.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab.
Entry for principal kudu/127.31.131.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab.
I20240507 02:11:07.483341 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:42727
--webserver_interface=127.31.131.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab
--principal=kudu/127.31.131.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:44393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.131.126:42727
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:11:07.560899 19002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:11:07.561048 19002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:11:07.561075 19002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:11:07.562304 19002 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240507 02:11:07.562343 19002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:11:07.562361 19002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:11:07.562376 19002 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240507 02:11:07.562390 19002 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240507 02:11:07.563920 19002 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:44393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.131.126:42727
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab
--principal=kudu/127.31.131.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:42727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.131.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:11:07.564102 19002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:11:07.564241 19002 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:11:07.567566 19010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:11:07.567545 19007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:07.567651 19002 server_base.cc:1034] running on GCE node
W20240507 02:11:07.567814 19008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:07.568025 19002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:11:07.568220 19002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:11:07.569337 19002 hybrid_clock.cc:648] HybridClock initialized: now 1715047867569325 us; error 26 us; skew 500 ppm
May 07 02:11:07 dist-test-slave-v0bg krb5kdc[18989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047867, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:11:07.571375 19002 init.cc:377] Logged in from keytab as kudu/127.31.131.126@KRBTEST.COM (short username kudu)
I20240507 02:11:07.571743 19002 webserver.cc:460] Webserver started at http://127.31.131.126:36329/ using document root <none> and password file <none>
I20240507 02:11:07.571892 19002 fs_manager.cc:359] Metadata directory not provided
I20240507 02:11:07.571929 19002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:11:07.572021 19002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:11:07.572830 19002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/minicluster-data/master-0/data/instance:
uuid: "e57407ea8a1c4fa78e1826b414d0fd3c"
format_stamp: "Formatted at 2024-05-07 02:11:07 on dist-test-slave-v0bg"
server_key: "ea785cca6e369bb6f0f921fc28e0abb3"
server_key_iv: "0fe8004af9b972bfb0c42546d812e1ba"
server_key_version: "encryptionkey@0"
I20240507 02:11:07.573086 19002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/minicluster-data/master-0/wal/instance:
uuid: "e57407ea8a1c4fa78e1826b414d0fd3c"
format_stamp: "Formatted at 2024-05-07 02:11:07 on dist-test-slave-v0bg"
server_key: "ea785cca6e369bb6f0f921fc28e0abb3"
server_key_iv: "0fe8004af9b972bfb0c42546d812e1ba"
server_key_version: "encryptionkey@0"
I20240507 02:11:07.574045 19002 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240507 02:11:07.574615 19017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:07.574749 19002 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240507 02:11:07.574815 19002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/minicluster-data/master-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/minicluster-data/master-0/wal
uuid: "e57407ea8a1c4fa78e1826b414d0fd3c"
format_stamp: "Formatted at 2024-05-07 02:11:07 on dist-test-slave-v0bg"
server_key: "ea785cca6e369bb6f0f921fc28e0abb3"
server_key_iv: "0fe8004af9b972bfb0c42546d812e1ba"
server_key_version: "encryptionkey@0"
I20240507 02:11:07.574872 19002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:11:07.611301 19002 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:11:07.612627 19002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:11:07.612721 19002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:11:07.615801 19002 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.126:42727
I20240507 02:11:07.615829 19069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.126:42727 every 8 connection(s)
I20240507 02:11:07.616065 19002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
I20240507 02:11:07.616772 19070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:11:07.618690 19070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e57407ea8a1c4fa78e1826b414d0fd3c: Bootstrap starting.
I20240507 02:11:07.619156 19070 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e57407ea8a1c4fa78e1826b414d0fd3c: Neither blocks nor log segments found. Creating new log.
I20240507 02:11:07.619418 19070 log.cc:824] T 00000000000000000000000000000000 P e57407ea8a1c4fa78e1826b414d0fd3c: Log is configured to *not* fsync() on all Append() calls
I20240507 02:11:07.619486 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 19002
I20240507 02:11:07.619592 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/minicluster-data/master-0/wal/instance
I20240507 02:11:07.619771 32269 external_mini_cluster.cc:1428] Setting key c05276e0441cb19cdad30bd602ca8199
I20240507 02:11:07.620049 19070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e57407ea8a1c4fa78e1826b414d0fd3c: No bootstrap required, opened a new log
I20240507 02:11:07.621500 19070 raft_consensus.cc:348] T 00000000000000000000000000000000 P e57407ea8a1c4fa78e1826b414d0fd3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e57407ea8a1c4fa78e1826b414d0fd3c" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 42727 } }
I20240507 02:11:07.621611 19070 raft_consensus.cc:374] T 00000000000000000000000000000000 P e57407ea8a1c4fa78e1826b414d0fd3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:11:07.621636 19070 raft_consensus.cc:729] T 00000000000000000000000000000000 P e57407ea8a1c4fa78e1826b414d0fd3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e57407ea8a1c4fa78e1826b414d0fd3c, State: Initialized, Role: FOLLOWER
I20240507 02:11:07.621748 19070 consensus_queue.cc:260] T 00000000000000000000000000000000 P e57407ea8a1c4fa78e1826b414d0fd3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e57407ea8a1c4fa78e1826b414d0fd3c" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 42727 } }
I20240507 02:11:07.621810 19070 raft_consensus.cc:388] T 00000000000000000000000000000000 P e57407ea8a1c4fa78e1826b414d0fd3c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 02:11:07.621841 19070 raft_consensus.cc:482] T 00000000000000000000000000000000 P e57407ea8a1c4fa78e1826b414d0fd3c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 02:11:07.621891 19070 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e57407ea8a1c4fa78e1826b414d0fd3c [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:11:07.622373 19070 raft_consensus.cc:504] T 00000000000000000000000000000000 P e57407ea8a1c4fa78e1826b414d0fd3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e57407ea8a1c4fa78e1826b414d0fd3c" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 42727 } }
I20240507 02:11:07.622463 19070 leader_election.cc:302] T 00000000000000000000000000000000 P e57407ea8a1c4fa78e1826b414d0fd3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e57407ea8a1c4fa78e1826b414d0fd3c; no voters:
I20240507 02:11:07.622637 19070 leader_election.cc:288] T 00000000000000000000000000000000 P e57407ea8a1c4fa78e1826b414d0fd3c [CANDIDATE]: Term 1 election: Requested vote from peers
I20240507 02:11:07.622735 19075 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e57407ea8a1c4fa78e1826b414d0fd3c [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:11:07.622848 19075 raft_consensus.cc:686] T 00000000000000000000000000000000 P e57407ea8a1c4fa78e1826b414d0fd3c [term 1 LEADER]: Becoming Leader. State: Replica: e57407ea8a1c4fa78e1826b414d0fd3c, State: Running, Role: LEADER
I20240507 02:11:07.622888 19070 sys_catalog.cc:564] T 00000000000000000000000000000000 P e57407ea8a1c4fa78e1826b414d0fd3c [sys.catalog]: configured and running, proceeding with master startup.
I20240507 02:11:07.623001 19075 consensus_queue.cc:237] T 00000000000000000000000000000000 P e57407ea8a1c4fa78e1826b414d0fd3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e57407ea8a1c4fa78e1826b414d0fd3c" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 42727 } }
I20240507 02:11:07.623297 19075 sys_catalog.cc:455] T 00000000000000000000000000000000 P e57407ea8a1c4fa78e1826b414d0fd3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e57407ea8a1c4fa78e1826b414d0fd3c. Latest consensus state: current_term: 1 leader_uuid: "e57407ea8a1c4fa78e1826b414d0fd3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e57407ea8a1c4fa78e1826b414d0fd3c" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 42727 } } }
I20240507 02:11:07.623394 19075 sys_catalog.cc:458] T 00000000000000000000000000000000 P e57407ea8a1c4fa78e1826b414d0fd3c [sys.catalog]: This master's current role is: LEADER
I20240507 02:11:07.623308 19076 sys_catalog.cc:455] T 00000000000000000000000000000000 P e57407ea8a1c4fa78e1826b414d0fd3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e57407ea8a1c4fa78e1826b414d0fd3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e57407ea8a1c4fa78e1826b414d0fd3c" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 42727 } } }
I20240507 02:11:07.623505 19076 sys_catalog.cc:458] T 00000000000000000000000000000000 P e57407ea8a1c4fa78e1826b414d0fd3c [sys.catalog]: This master's current role is: LEADER
W20240507 02:11:07.624665 19091 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e57407ea8a1c4fa78e1826b414d0fd3c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240507 02:11:07.624717 19091 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240507 02:11:07.624768 19081 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 02:11:07.624925 19081 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240507 02:11:07.626171 19081 catalog_manager.cc:1347] Generated new cluster ID: 7cb1f950c24c40f28027969bca4cfecb
I20240507 02:11:07.626237 19081 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
May 07 02:11:07 dist-test-slave-v0bg krb5kdc[18989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047867, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:11:07.629473 19074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:07.620174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55228 (local address 127.31.131.126:42727)
0507 02:11:07.620331 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:07.620345 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:07.620363 (+ 18us) server_negotiation.cc:407] Connection header received
0507 02:11:07.621473 (+ 1110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:07.621477 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:07.621512 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:07.621618 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:07.622321 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:07.623644 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:07.624609 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:07.624818 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:07.626959 (+ 2141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:07.626964 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:07.626964 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:11:07.626983 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:07.628414 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:07.628723 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:07.628725 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:07.628725 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:07.628759 (+ 34us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:07.628876 (+ 117us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:07.628877 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:07.628877 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:07.629029 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:07.629126 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:07.629218 (+ 92us) server_negotiation.cc:299] Negotiation successful
0507 02:11:07.629303 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":80,"threads_started":1}
I20240507 02:11:07.637109 19081 catalog_manager.cc:1370] Generated new certificate authority record
I20240507 02:11:07.637568 19081 catalog_manager.cc:1504] Loading token signing keys...
I20240507 02:11:07.643105 19081 catalog_manager.cc:5857] T 00000000000000000000000000000000 P e57407ea8a1c4fa78e1826b414d0fd3c: Generated new TSK 0
I20240507 02:11:07.643256 19081 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.131.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab.
Entry for principal kudu/127.31.131.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab.
I20240507 02:11:07.674353 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.65:0
--local_ip_for_outbound_sockets=127.31.131.65
--webserver_interface=127.31.131.65
--webserver_port=0
--tserver_master_addrs=127.31.131.126:42727
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab
--principal=kudu/127.31.131.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:44393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:11:07.765691 19096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:11:07.765856 19096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:11:07.765885 19096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:11:07.767163 19096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:11:07.767204 19096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:11:07.767239 19096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.65
I20240507 02:11:07.768715 19096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:44393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.65.keytab
--principal=kudu/127.31.131.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.131.65
--webserver_port=0
--tserver_master_addrs=127.31.131.126:42727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.65
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:11:07.768908 19096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:11:07.769071 19096 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:11:07.773072 19101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:11:07.773088 19104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:11:07.773066 19102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:07.773442 19096 server_base.cc:1034] running on GCE node
I20240507 02:11:07.773610 19096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:11:07.773816 19096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:11:07.774969 19096 hybrid_clock.cc:648] HybridClock initialized: now 1715047867774958 us; error 28 us; skew 500 ppm
May 07 02:11:07 dist-test-slave-v0bg krb5kdc[18989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047867, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:11:07.777508 19096 init.cc:377] Logged in from keytab as kudu/127.31.131.65@KRBTEST.COM (short username kudu)
I20240507 02:11:07.777876 19096 webserver.cc:460] Webserver started at http://127.31.131.65:45159/ using document root <none> and password file <none>
I20240507 02:11:07.778028 19096 fs_manager.cc:359] Metadata directory not provided
I20240507 02:11:07.778067 19096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:11:07.778185 19096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:11:07.779084 19096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/minicluster-data/ts-0/data/instance:
uuid: "f014460be902419ebb83e436e3b0f52e"
format_stamp: "Formatted at 2024-05-07 02:11:07 on dist-test-slave-v0bg"
server_key: "3ea1f1cef0a83f5337b99ac1d6101064"
server_key_iv: "018f6be8f30f62be6807cdaf8ba0288e"
server_key_version: "encryptionkey@0"
I20240507 02:11:07.779342 19096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/minicluster-data/ts-0/wal/instance:
uuid: "f014460be902419ebb83e436e3b0f52e"
format_stamp: "Formatted at 2024-05-07 02:11:07 on dist-test-slave-v0bg"
server_key: "3ea1f1cef0a83f5337b99ac1d6101064"
server_key_iv: "018f6be8f30f62be6807cdaf8ba0288e"
server_key_version: "encryptionkey@0"
I20240507 02:11:07.780380 19096 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240507 02:11:07.780982 19111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:07.781162 19096 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240507 02:11:07.781230 19096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/minicluster-data/ts-0/wal
uuid: "f014460be902419ebb83e436e3b0f52e"
format_stamp: "Formatted at 2024-05-07 02:11:07 on dist-test-slave-v0bg"
server_key: "3ea1f1cef0a83f5337b99ac1d6101064"
server_key_iv: "018f6be8f30f62be6807cdaf8ba0288e"
server_key_version: "encryptionkey@0"
I20240507 02:11:07.781291 19096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:11:07.793159 19096 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:11:07.794884 19096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:11:07.794978 19096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:11:07.795131 19096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:11:07.795367 19096 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:11:07.795398 19096 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:07.795423 19096 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:11:07.795452 19096 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:07.800204 19096 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.65:33705
I20240507 02:11:07.800264 19225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.65:33705 every 8 connection(s)
I20240507 02:11:07.800468 19096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
May 07 02:11:07 dist-test-slave-v0bg krb5kdc[18989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047867, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.65@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:11:07.807133 19074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:07.801009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:60617 (local address 127.31.131.126:42727)
0507 02:11:07.801052 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:07.801053 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:07.801228 (+ 175us) server_negotiation.cc:407] Connection header received
0507 02:11:07.801642 (+ 414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:07.801643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:07.801655 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:07.801710 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:07.802601 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:07.803057 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:07.803556 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:07.803716 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:07.805284 (+ 1568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:07.805287 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:07.805287 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:11:07.805293 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:07.806303 (+ 1010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:07.806594 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:07.806595 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:07.806595 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:07.806613 (+ 18us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:07.806731 (+ 118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:07.806731 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:07.806732 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:07.806802 (+ 70us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:07.806841 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:07.807013 (+ 172us) server_negotiation.cc:299] Negotiation successful
0507 02:11:07.807073 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20240507 02:11:07.807209 19228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:07.800988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:42727 (local address 127.31.131.65:60617)
0507 02:11:07.801138 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:07.801160 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0507 02:11:07.801558 (+ 398us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:11:07.801764 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:11:07.801768 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:11:07.802011 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:11:07.802494 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:07.802498 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:07.803129 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:11:07.803130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:11:07.803487 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:07.803489 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:07.803602 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:07.803997 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:11:07.804009 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:11:07.805206 (+ 1197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:11:07.806365 (+ 1159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:11:07.806369 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:11:07.806371 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:11:07.806532 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:11:07.806658 (+ 126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:11:07.806659 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:11:07.806659 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:11:07.806701 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:11:07.806840 (+ 139us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:11:07.806842 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:11:07.806926 (+ 84us) client_negotiation.cc:769] Sending connection context
0507 02:11:07.806997 (+ 71us) client_negotiation.cc:240] Negotiation successful
0507 02:11:07.807088 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":87,"threads_started":1}
I20240507 02:11:07.807634 19226 heartbeater.cc:347] Connected to a master server at 127.31.131.126:42727
I20240507 02:11:07.807705 19226 heartbeater.cc:464] Registering TS with master...
I20240507 02:11:07.807879 19226 heartbeater.cc:511] Master 127.31.131.126:42727 requested a full tablet report, sending...
I20240507 02:11:07.807984 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 19096
I20240507 02:11:07.808089 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/minicluster-data/ts-0/wal/instance
I20240507 02:11:07.808264 32269 external_mini_cluster.cc:1428] Setting key 148bdbe4da8215791d93b0ebfc3a3a4e
I20240507 02:11:07.808323 19034 ts_manager.cc:194] Registered new tserver with Master: f014460be902419ebb83e436e3b0f52e (127.31.131.65:33705)
I20240507 02:11:07.808981 19034 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.131.65@KRBTEST.COM'} at 127.31.131.65:60617
WARNING: no policy specified for kudu/127.31.131.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab.
Entry for principal kudu/127.31.131.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab.
I20240507 02:11:07.828477 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.66:0
--local_ip_for_outbound_sockets=127.31.131.66
--webserver_interface=127.31.131.66
--webserver_port=0
--tserver_master_addrs=127.31.131.126:42727
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab
--principal=kudu/127.31.131.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:44393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:11:07.901345 19231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:11:07.901502 19231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:11:07.901522 19231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:11:07.902761 19231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:11:07.902797 19231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:11:07.902819 19231 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.66
I20240507 02:11:07.904240 19231 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:44393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.66.keytab
--principal=kudu/127.31.131.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.131.66
--webserver_port=0
--tserver_master_addrs=127.31.131.126:42727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19231
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.66
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:11:07.904384 19231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:11:07.904521 19231 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240507 02:11:07.907923 19231 server_base.cc:1034] running on GCE node
W20240507 02:11:07.907946 19237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:11:07.908068 19239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:11:07.908000 19236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:07.908252 19231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:11:07.908416 19231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:11:07.909535 19231 hybrid_clock.cc:648] HybridClock initialized: now 1715047867909518 us; error 31 us; skew 500 ppm
May 07 02:11:07 dist-test-slave-v0bg krb5kdc[18989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047867, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:11:07.911557 19231 init.cc:377] Logged in from keytab as kudu/127.31.131.66@KRBTEST.COM (short username kudu)
I20240507 02:11:07.911913 19231 webserver.cc:460] Webserver started at http://127.31.131.66:37569/ using document root <none> and password file <none>
I20240507 02:11:07.912075 19231 fs_manager.cc:359] Metadata directory not provided
I20240507 02:11:07.912122 19231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:11:07.912212 19231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:11:07.913040 19231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/minicluster-data/ts-1/data/instance:
uuid: "7c4cedcfe9a64d9d83e9b721e5c83606"
format_stamp: "Formatted at 2024-05-07 02:11:07 on dist-test-slave-v0bg"
server_key: "faab946fc4e0f7697754200991eb6303"
server_key_iv: "25634701d3fb911589831d4581ef9ebc"
server_key_version: "encryptionkey@0"
I20240507 02:11:07.913297 19231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/minicluster-data/ts-1/wal/instance:
uuid: "7c4cedcfe9a64d9d83e9b721e5c83606"
format_stamp: "Formatted at 2024-05-07 02:11:07 on dist-test-slave-v0bg"
server_key: "faab946fc4e0f7697754200991eb6303"
server_key_iv: "25634701d3fb911589831d4581ef9ebc"
server_key_version: "encryptionkey@0"
I20240507 02:11:07.914292 19231 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240507 02:11:07.914865 19246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:07.914991 19231 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:07.915055 19231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/minicluster-data/ts-1/wal
uuid: "7c4cedcfe9a64d9d83e9b721e5c83606"
format_stamp: "Formatted at 2024-05-07 02:11:07 on dist-test-slave-v0bg"
server_key: "faab946fc4e0f7697754200991eb6303"
server_key_iv: "25634701d3fb911589831d4581ef9ebc"
server_key_version: "encryptionkey@0"
I20240507 02:11:07.915109 19231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:11:07.936115 19231 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:11:07.937427 19231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:11:07.937520 19231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:11:07.937673 19231 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:11:07.937906 19231 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:11:07.937935 19231 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:07.937960 19231 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:11:07.937975 19231 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:07.942665 19231 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.66:34389
I20240507 02:11:07.942727 19359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.66:34389 every 8 connection(s)
I20240507 02:11:07.942934 19231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
May 07 02:11:07 dist-test-slave-v0bg krb5kdc[18989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047867, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.66@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:11:07.949527 19074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:07.943502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.66:50925 (local address 127.31.131.126:42727)
0507 02:11:07.943556 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:07.943558 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:07.943677 (+ 119us) server_negotiation.cc:407] Connection header received
0507 02:11:07.944074 (+ 397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:07.944075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:07.944083 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:07.944134 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:07.944954 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:07.945423 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:07.945883 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:07.945981 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:07.947578 (+ 1597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:07.947582 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:07.947582 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:11:07.947588 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:07.948746 (+ 1158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:07.949006 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:07.949007 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:07.949007 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:07.949023 (+ 16us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:07.949134 (+ 111us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:07.949135 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:07.949135 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:07.949208 (+ 73us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:07.949246 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:07.949409 (+ 163us) server_negotiation.cc:299] Negotiation successful
0507 02:11:07.949465 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20240507 02:11:07.949610 19362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:07.943454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:42727 (local address 127.31.131.66:50925)
0507 02:11:07.943592 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:07.943618 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0507 02:11:07.943994 (+ 376us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:11:07.944185 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:11:07.944188 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:11:07.944430 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:11:07.944890 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:07.944893 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:07.945481 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:11:07.945482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:11:07.945832 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:07.945833 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:07.945916 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:07.946314 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:11:07.946326 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:11:07.947501 (+ 1175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:11:07.948806 (+ 1305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:11:07.948809 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:11:07.948811 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:11:07.948955 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:11:07.949064 (+ 109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:11:07.949064 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:11:07.949065 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:11:07.949104 (+ 39us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:11:07.949245 (+ 141us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:11:07.949247 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:11:07.949330 (+ 83us) client_negotiation.cc:769] Sending connection context
0507 02:11:07.949400 (+ 70us) client_negotiation.cc:240] Negotiation successful
0507 02:11:07.949485 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":69,"threads_started":1}
I20240507 02:11:07.950042 19360 heartbeater.cc:347] Connected to a master server at 127.31.131.126:42727
I20240507 02:11:07.950111 19360 heartbeater.cc:464] Registering TS with master...
I20240507 02:11:07.950254 19360 heartbeater.cc:511] Master 127.31.131.126:42727 requested a full tablet report, sending...
I20240507 02:11:07.950616 19034 ts_manager.cc:194] Registered new tserver with Master: 7c4cedcfe9a64d9d83e9b721e5c83606 (127.31.131.66:34389)
I20240507 02:11:07.950942 19034 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.131.66@KRBTEST.COM'} at 127.31.131.66:50925
I20240507 02:11:07.951756 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 19231
I20240507 02:11:07.951841 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/minicluster-data/ts-1/wal/instance
I20240507 02:11:07.951969 32269 external_mini_cluster.cc:1428] Setting key d081be45eecadd435d7e0a23bbc14929
WARNING: no policy specified for kudu/127.31.131.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab.
Entry for principal kudu/127.31.131.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab.
I20240507 02:11:07.970610 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.67:0
--local_ip_for_outbound_sockets=127.31.131.67
--webserver_interface=127.31.131.67
--webserver_port=0
--tserver_master_addrs=127.31.131.126:42727
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab
--principal=kudu/127.31.131.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:44393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:11:08.043808 19365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:11:08.043982 19365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:11:08.044013 19365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:11:08.045256 19365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:11:08.045300 19365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:11:08.045336 19365 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.67
I20240507 02:11:08.046824 19365 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:44393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.67.keytab
--principal=kudu/127.31.131.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.131.67
--webserver_port=0
--tserver_master_addrs=127.31.131.126:42727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.67
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:11:08.047013 19365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:11:08.047168 19365 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:11:08.050657 19370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:08.050717 19365 server_base.cc:1034] running on GCE node
W20240507 02:11:08.050765 19373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:11:08.050783 19371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:08.051067 19365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:11:08.051262 19365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:11:08.052397 19365 hybrid_clock.cc:648] HybridClock initialized: now 1715047868052381 us; error 28 us; skew 500 ppm
May 07 02:11:08 dist-test-slave-v0bg krb5kdc[18989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047868, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:11:08.054535 19365 init.cc:377] Logged in from keytab as kudu/127.31.131.67@KRBTEST.COM (short username kudu)
I20240507 02:11:08.054930 19365 webserver.cc:460] Webserver started at http://127.31.131.67:37875/ using document root <none> and password file <none>
I20240507 02:11:08.055081 19365 fs_manager.cc:359] Metadata directory not provided
I20240507 02:11:08.055123 19365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:11:08.055224 19365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:11:08.056062 19365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/minicluster-data/ts-2/data/instance:
uuid: "10f4e07ba6584754b3d0e611cf3b9fa3"
format_stamp: "Formatted at 2024-05-07 02:11:08 on dist-test-slave-v0bg"
server_key: "3f063acf58d2fc94c82cec968d65de69"
server_key_iv: "6a2ec960fcfeac0b01216a4fe6e6ba38"
server_key_version: "encryptionkey@0"
I20240507 02:11:08.056341 19365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/minicluster-data/ts-2/wal/instance:
uuid: "10f4e07ba6584754b3d0e611cf3b9fa3"
format_stamp: "Formatted at 2024-05-07 02:11:08 on dist-test-slave-v0bg"
server_key: "3f063acf58d2fc94c82cec968d65de69"
server_key_iv: "6a2ec960fcfeac0b01216a4fe6e6ba38"
server_key_version: "encryptionkey@0"
I20240507 02:11:08.057379 19365 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240507 02:11:08.057952 19380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:08.058092 19365 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:08.058161 19365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/minicluster-data/ts-2/wal
uuid: "10f4e07ba6584754b3d0e611cf3b9fa3"
format_stamp: "Formatted at 2024-05-07 02:11:08 on dist-test-slave-v0bg"
server_key: "3f063acf58d2fc94c82cec968d65de69"
server_key_iv: "6a2ec960fcfeac0b01216a4fe6e6ba38"
server_key_version: "encryptionkey@0"
I20240507 02:11:08.058220 19365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:11:08.067056 19365 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:11:08.068814 19365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:11:08.068915 19365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:11:08.069074 19365 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:11:08.069319 19365 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:11:08.069351 19365 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:08.069379 19365 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:11:08.069407 19365 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:08.074733 19365 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.67:37049
I20240507 02:11:08.074769 19493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.67:37049 every 8 connection(s)
I20240507 02:11:08.074995 19365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
May 07 02:11:08 dist-test-slave-v0bg krb5kdc[18989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047868, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.67@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:11:08.081905 19074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:08.075639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.67:52273 (local address 127.31.131.126:42727)
0507 02:11:08.075680 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:08.075682 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:08.075856 (+ 174us) server_negotiation.cc:407] Connection header received
0507 02:11:08.076253 (+ 397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:08.076254 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:08.076266 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:08.076318 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:08.077195 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:08.077665 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:08.078197 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:08.078359 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:08.079988 (+ 1629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:08.079991 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:08.079992 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:11:08.079998 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:08.081050 (+ 1052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:08.081348 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:08.081348 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:08.081349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:08.081363 (+ 14us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:08.081499 (+ 136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:08.081499 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:08.081499 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:08.081570 (+ 71us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:08.081612 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:08.081788 (+ 176us) server_negotiation.cc:299] Negotiation successful
0507 02:11:08.081845 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20240507 02:11:08.081981 19496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:08.075611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:42727 (local address 127.31.131.67:52273)
0507 02:11:08.075767 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:08.075788 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0507 02:11:08.076166 (+ 378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:11:08.076369 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:11:08.076372 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:11:08.076610 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:11:08.077112 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:08.077116 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:08.077737 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:11:08.077738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:11:08.078130 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:08.078131 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:08.078242 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:08.078686 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:11:08.078699 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:11:08.079910 (+ 1211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:11:08.081113 (+ 1203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:11:08.081118 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:11:08.081120 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:11:08.081292 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:11:08.081406 (+ 114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:11:08.081407 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:11:08.081407 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:11:08.081449 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:11:08.081611 (+ 162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:11:08.081613 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:11:08.081697 (+ 84us) client_negotiation.cc:769] Sending connection context
0507 02:11:08.081773 (+ 76us) client_negotiation.cc:240] Negotiation successful
0507 02:11:08.081860 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":77,"threads_started":1}
I20240507 02:11:08.082402 19494 heartbeater.cc:347] Connected to a master server at 127.31.131.126:42727
I20240507 02:11:08.082473 19494 heartbeater.cc:464] Registering TS with master...
I20240507 02:11:08.082666 19494 heartbeater.cc:511] Master 127.31.131.126:42727 requested a full tablet report, sending...
I20240507 02:11:08.083005 19034 ts_manager.cc:194] Registered new tserver with Master: 10f4e07ba6584754b3d0e611cf3b9fa3 (127.31.131.67:37049)
I20240507 02:11:08.083353 19034 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.131.67@KRBTEST.COM'} at 127.31.131.67:52273
I20240507 02:11:08.083520 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 19365
I20240507 02:11:08.083608 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/minicluster-data/ts-2/wal/instance
I20240507 02:11:08.083756 32269 external_mini_cluster.cc:1428] Setting key 152c10e572f8d6bee206c6bca74ff443
I20240507 02:11:08.084565 32269 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240507 02:11:08.090441 19074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:08.085702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55236 (local address 127.31.131.126:42727)
0507 02:11:08.085750 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:08.085752 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:08.085864 (+ 112us) server_negotiation.cc:407] Connection header received
0507 02:11:08.085930 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:08.085931 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:08.085940 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:08.085996 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:08.086666 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:08.087104 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:08.087632 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:08.087782 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:08.088570 (+ 788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:08.088573 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:08.088573 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:11:08.088579 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:08.089545 (+ 966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:08.089819 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:08.089820 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:08.089820 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:08.089842 (+ 22us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:08.089987 (+ 145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:08.089987 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:08.089988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:08.090084 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:08.090172 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:08.090281 (+ 109us) server_negotiation.cc:299] Negotiation successful
0507 02:11:08.090356 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20240507 02:11:08.091758 19034 catalog_manager.cc:1904] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55236:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 02:11:08.093055 19034 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 02:11:08.097707 19502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:08.094864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.65:33705 (local address 127.0.0.1:45620)
0507 02:11:08.095078 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:08.095101 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0507 02:11:08.095176 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:11:08.095461 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:11:08.095464 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:11:08.095467 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:11:08.095680 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:08.095683 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:08.096635 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:11:08.096636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:11:08.097371 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:08.097373 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:08.097451 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:08.097483 (+ 32us) client_negotiation.cc:769] Sending connection context
0507 02:11:08.097535 (+ 52us) client_negotiation.cc:240] Negotiation successful
0507 02:11:08.097570 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":57,"threads_started":1}
I20240507 02:11:08.098039 19506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:08.095060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.66:34389 (local address 127.0.0.1:60506)
0507 02:11:08.095286 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:08.095294 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0507 02:11:08.095335 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:11:08.095518 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:11:08.095519 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:11:08.095521 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:11:08.095791 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:08.095794 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:08.097107 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:11:08.097108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:11:08.097823 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:08.097825 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:08.097889 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:08.097893 (+ 4us) client_negotiation.cc:769] Sending connection context
0507 02:11:08.097922 (+ 29us) client_negotiation.cc:240] Negotiation successful
0507 02:11:08.097965 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":71,"threads_started":1}
I20240507 02:11:08.098413 19505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:08.094937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45620 (local address 127.31.131.65:33705)
0507 02:11:08.095104 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:08.095114 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:08.095154 (+ 40us) server_negotiation.cc:407] Connection header received
0507 02:11:08.095256 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:08.095259 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:08.095276 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:08.095363 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:11:08.095775 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:08.096557 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:08.097480 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:08.098281 (+ 801us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:08.098312 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:08.098335 (+ 23us) server_negotiation.cc:299] Negotiation successful
0507 02:11:08.098361 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":71,"threads_started":1}
I20240507 02:11:08.098678 19507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:08.095140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60506 (local address 127.31.131.66:34389)
0507 02:11:08.095313 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:08.095320 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:08.095330 (+ 10us) server_negotiation.cc:407] Connection header received
0507 02:11:08.095398 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:08.095399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:08.095415 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:08.095498 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:11:08.095869 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:08.097037 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:08.097920 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:08.098455 (+ 535us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:08.098495 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:08.098527 (+ 32us) server_negotiation.cc:299] Negotiation successful
0507 02:11:08.098601 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":82,"threads_started":1}
I20240507 02:11:08.098903 19159 tablet_service.cc:1467] Processing CreateTablet for tablet 1d9f865d67944d5e9786967f92c60607 (DEFAULT_TABLE table=test-table [id=e70927d60db646d6be1cfa3efc24eb73]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:11:08.099129 19159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d9f865d67944d5e9786967f92c60607. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:11:08.099135 19294 tablet_service.cc:1467] Processing CreateTablet for tablet 1d9f865d67944d5e9786967f92c60607 (DEFAULT_TABLE table=test-table [id=e70927d60db646d6be1cfa3efc24eb73]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:11:08.099392 19294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d9f865d67944d5e9786967f92c60607. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:11:08.101363 19508 tablet_bootstrap.cc:492] T 1d9f865d67944d5e9786967f92c60607 P f014460be902419ebb83e436e3b0f52e: Bootstrap starting.
I20240507 02:11:08.101672 19509 tablet_bootstrap.cc:492] T 1d9f865d67944d5e9786967f92c60607 P 7c4cedcfe9a64d9d83e9b721e5c83606: Bootstrap starting.
I20240507 02:11:08.101754 19503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:08.094864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.67:37049 (local address 127.0.0.1:53748)
0507 02:11:08.095275 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:08.095283 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0507 02:11:08.095330 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:11:08.095582 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:11:08.095583 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:11:08.095585 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:11:08.095770 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:08.095772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:08.096993 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:11:08.096994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:11:08.098000 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:08.098003 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:08.098073 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:08.098780 (+ 707us) client_negotiation.cc:769] Sending connection context
0507 02:11:08.098844 (+ 64us) client_negotiation.cc:240] Negotiation successful
0507 02:11:08.101692 (+ 2848us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"mutex_wait_us":3,"thread_start_us":50,"threads_started":1}
I20240507 02:11:08.101878 19508 tablet_bootstrap.cc:654] T 1d9f865d67944d5e9786967f92c60607 P f014460be902419ebb83e436e3b0f52e: Neither blocks nor log segments found. Creating new log.
I20240507 02:11:08.102053 19504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:08.094904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53748 (local address 127.31.131.67:37049)
0507 02:11:08.095418 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:08.095424 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:08.095432 (+ 8us) server_negotiation.cc:407] Connection header received
0507 02:11:08.095458 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:08.095459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:08.095471 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:08.095534 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:11:08.095841 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:08.096914 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:08.098237 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:08.098726 (+ 489us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:08.098762 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:08.101942 (+ 3180us) server_negotiation.cc:299] Negotiation successful
0507 02:11:08.101987 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":69,"threads_started":1}
I20240507 02:11:08.102121 19508 log.cc:824] T 1d9f865d67944d5e9786967f92c60607 P f014460be902419ebb83e436e3b0f52e: Log is configured to *not* fsync() on all Append() calls
I20240507 02:11:08.102196 19509 tablet_bootstrap.cc:654] T 1d9f865d67944d5e9786967f92c60607 P 7c4cedcfe9a64d9d83e9b721e5c83606: Neither blocks nor log segments found. Creating new log.
I20240507 02:11:08.102453 19509 log.cc:824] T 1d9f865d67944d5e9786967f92c60607 P 7c4cedcfe9a64d9d83e9b721e5c83606: Log is configured to *not* fsync() on all Append() calls
I20240507 02:11:08.102545 19428 tablet_service.cc:1467] Processing CreateTablet for tablet 1d9f865d67944d5e9786967f92c60607 (DEFAULT_TABLE table=test-table [id=e70927d60db646d6be1cfa3efc24eb73]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:11:08.102809 19428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d9f865d67944d5e9786967f92c60607. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:11:08.102924 19508 tablet_bootstrap.cc:492] T 1d9f865d67944d5e9786967f92c60607 P f014460be902419ebb83e436e3b0f52e: No bootstrap required, opened a new log
I20240507 02:11:08.103030 19508 ts_tablet_manager.cc:1367] T 1d9f865d67944d5e9786967f92c60607 P f014460be902419ebb83e436e3b0f52e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240507 02:11:08.103196 19509 tablet_bootstrap.cc:492] T 1d9f865d67944d5e9786967f92c60607 P 7c4cedcfe9a64d9d83e9b721e5c83606: No bootstrap required, opened a new log
I20240507 02:11:08.103268 19509 ts_tablet_manager.cc:1367] T 1d9f865d67944d5e9786967f92c60607 P 7c4cedcfe9a64d9d83e9b721e5c83606: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240507 02:11:08.104566 19508 raft_consensus.cc:348] T 1d9f865d67944d5e9786967f92c60607 P f014460be902419ebb83e436e3b0f52e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f4e07ba6584754b3d0e611cf3b9fa3" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 37049 } } peers { permanent_uuid: "f014460be902419ebb83e436e3b0f52e" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33705 } } peers { permanent_uuid: "7c4cedcfe9a64d9d83e9b721e5c83606" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 34389 } }
I20240507 02:11:08.104671 19508 raft_consensus.cc:374] T 1d9f865d67944d5e9786967f92c60607 P f014460be902419ebb83e436e3b0f52e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:11:08.104691 19508 raft_consensus.cc:729] T 1d9f865d67944d5e9786967f92c60607 P f014460be902419ebb83e436e3b0f52e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f014460be902419ebb83e436e3b0f52e, State: Initialized, Role: FOLLOWER
I20240507 02:11:08.104835 19508 consensus_queue.cc:260] T 1d9f865d67944d5e9786967f92c60607 P f014460be902419ebb83e436e3b0f52e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f4e07ba6584754b3d0e611cf3b9fa3" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 37049 } } peers { permanent_uuid: "f014460be902419ebb83e436e3b0f52e" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33705 } } peers { permanent_uuid: "7c4cedcfe9a64d9d83e9b721e5c83606" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 34389 } }
I20240507 02:11:08.105052 19508 ts_tablet_manager.cc:1398] T 1d9f865d67944d5e9786967f92c60607 P f014460be902419ebb83e436e3b0f52e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240507 02:11:08.105139 19512 raft_consensus.cc:482] T 1d9f865d67944d5e9786967f92c60607 P f014460be902419ebb83e436e3b0f52e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 02:11:08.105191 19512 raft_consensus.cc:504] T 1d9f865d67944d5e9786967f92c60607 P f014460be902419ebb83e436e3b0f52e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f4e07ba6584754b3d0e611cf3b9fa3" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 37049 } } peers { permanent_uuid: "f014460be902419ebb83e436e3b0f52e" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33705 } } peers { permanent_uuid: "7c4cedcfe9a64d9d83e9b721e5c83606" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 34389 } }
I20240507 02:11:08.105368 19509 raft_consensus.cc:348] T 1d9f865d67944d5e9786967f92c60607 P 7c4cedcfe9a64d9d83e9b721e5c83606 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f4e07ba6584754b3d0e611cf3b9fa3" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 37049 } } peers { permanent_uuid: "f014460be902419ebb83e436e3b0f52e" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33705 } } peers { permanent_uuid: "7c4cedcfe9a64d9d83e9b721e5c83606" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 34389 } }
I20240507 02:11:08.105494 19509 raft_consensus.cc:374] T 1d9f865d67944d5e9786967f92c60607 P 7c4cedcfe9a64d9d83e9b721e5c83606 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:11:08.105520 19509 raft_consensus.cc:729] T 1d9f865d67944d5e9786967f92c60607 P 7c4cedcfe9a64d9d83e9b721e5c83606 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c4cedcfe9a64d9d83e9b721e5c83606, State: Initialized, Role: FOLLOWER
I20240507 02:11:08.105746 19226 heartbeater.cc:503] Master 127.31.131.126:42727 was elected leader, sending a full tablet report...
I20240507 02:11:08.105932 19512 leader_election.cc:288] T 1d9f865d67944d5e9786967f92c60607 P f014460be902419ebb83e436e3b0f52e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10f4e07ba6584754b3d0e611cf3b9fa3 (127.31.131.67:37049), 7c4cedcfe9a64d9d83e9b721e5c83606 (127.31.131.66:34389)
I20240507 02:11:08.106132 19509 consensus_queue.cc:260] T 1d9f865d67944d5e9786967f92c60607 P 7c4cedcfe9a64d9d83e9b721e5c83606 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f4e07ba6584754b3d0e611cf3b9fa3" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 37049 } } peers { permanent_uuid: "f014460be902419ebb83e436e3b0f52e" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33705 } } peers { permanent_uuid: "7c4cedcfe9a64d9d83e9b721e5c83606" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 34389 } }
I20240507 02:11:08.106366 19509 ts_tablet_manager.cc:1398] T 1d9f865d67944d5e9786967f92c60607 P 7c4cedcfe9a64d9d83e9b721e5c83606: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240507 02:11:08.106463 19360 heartbeater.cc:503] Master 127.31.131.126:42727 was elected leader, sending a full tablet report...
I20240507 02:11:08.107468 19514 tablet_bootstrap.cc:492] T 1d9f865d67944d5e9786967f92c60607 P 10f4e07ba6584754b3d0e611cf3b9fa3: Bootstrap starting.
I20240507 02:11:08.107972 19514 tablet_bootstrap.cc:654] T 1d9f865d67944d5e9786967f92c60607 P 10f4e07ba6584754b3d0e611cf3b9fa3: Neither blocks nor log segments found. Creating new log.
I20240507 02:11:08.108068 19228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:08.105531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.66:34389 (local address 127.31.131.65:56957)
0507 02:11:08.105615 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:08.105623 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0507 02:11:08.105660 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:11:08.105820 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:11:08.105821 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:11:08.105823 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:11:08.106030 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:08.106031 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:08.107275 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:11:08.107276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:11:08.107875 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:08.107877 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:08.107939 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:08.107942 (+ 3us) client_negotiation.cc:769] Sending connection context
0507 02:11:08.107969 (+ 27us) client_negotiation.cc:240] Negotiation successful
0507 02:11:08.108011 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20240507 02:11:08.108263 19514 log.cc:824] T 1d9f865d67944d5e9786967f92c60607 P 10f4e07ba6584754b3d0e611cf3b9fa3: Log is configured to *not* fsync() on all Append() calls
I20240507 02:11:08.108471 19507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:08.105617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:56957 (local address 127.31.131.66:34389)
0507 02:11:08.105653 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:08.105655 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:08.105665 (+ 10us) server_negotiation.cc:407] Connection header received
0507 02:11:08.105710 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:08.105711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:08.105719 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:08.105771 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:11:08.106106 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:08.107205 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:08.107947 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:08.108374 (+ 427us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:08.108381 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:08.108399 (+ 18us) server_negotiation.cc:299] Negotiation successful
0507 02:11:08.108426 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20240507 02:11:08.108683 19314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d9f865d67944d5e9786967f92c60607" candidate_uuid: "f014460be902419ebb83e436e3b0f52e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c4cedcfe9a64d9d83e9b721e5c83606" is_pre_election: true
I20240507 02:11:08.108780 19314 raft_consensus.cc:2454] T 1d9f865d67944d5e9786967f92c60607 P 7c4cedcfe9a64d9d83e9b721e5c83606 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f014460be902419ebb83e436e3b0f52e in term 0.
I20240507 02:11:08.108867 19515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:08.105536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.67:37049 (local address 127.31.131.65:40541)
0507 02:11:08.105884 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:08.105893 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0507 02:11:08.105963 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:11:08.106126 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:11:08.106127 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:11:08.106129 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:11:08.106359 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:08.106362 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:08.107237 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:11:08.107238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:11:08.108072 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:08.108075 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:08.108660 (+ 585us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:08.108663 (+ 3us) client_negotiation.cc:769] Sending connection context
0507 02:11:08.108755 (+ 92us) client_negotiation.cc:240] Negotiation successful
0507 02:11:08.108793 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"mutex_wait_us":6,"thread_start_us":66,"threads_started":1}
I20240507 02:11:08.108894 19514 tablet_bootstrap.cc:492] T 1d9f865d67944d5e9786967f92c60607 P 10f4e07ba6584754b3d0e611cf3b9fa3: No bootstrap required, opened a new log
I20240507 02:11:08.108945 19514 ts_tablet_manager.cc:1367] T 1d9f865d67944d5e9786967f92c60607 P 10f4e07ba6584754b3d0e611cf3b9fa3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240507 02:11:08.108968 19115 leader_election.cc:302] T 1d9f865d67944d5e9786967f92c60607 P f014460be902419ebb83e436e3b0f52e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c4cedcfe9a64d9d83e9b721e5c83606, f014460be902419ebb83e436e3b0f52e; no voters:
I20240507 02:11:08.109109 19504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:08.105678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:40541 (local address 127.31.131.67:37049)
0507 02:11:08.105823 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:08.105825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:08.105942 (+ 117us) server_negotiation.cc:407] Connection header received
0507 02:11:08.106031 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:08.106032 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:08.106040 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:08.106068 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:11:08.106458 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:08.107121 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:08.108183 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:08.108622 (+ 439us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:08.108629 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:08.108862 (+ 233us) server_negotiation.cc:299] Negotiation successful
0507 02:11:08.109056 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
I20240507 02:11:08.109124 19513 raft_consensus.cc:2781] T 1d9f865d67944d5e9786967f92c60607 P f014460be902419ebb83e436e3b0f52e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 02:11:08.109189 19513 raft_consensus.cc:482] T 1d9f865d67944d5e9786967f92c60607 P f014460be902419ebb83e436e3b0f52e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 02:11:08.109340 19513 raft_consensus.cc:3037] T 1d9f865d67944d5e9786967f92c60607 P f014460be902419ebb83e436e3b0f52e [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:11:08.109294 19448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d9f865d67944d5e9786967f92c60607" candidate_uuid: "f014460be902419ebb83e436e3b0f52e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f4e07ba6584754b3d0e611cf3b9fa3" is_pre_election: true
I20240507 02:11:08.110085 19513 raft_consensus.cc:504] T 1d9f865d67944d5e9786967f92c60607 P f014460be902419ebb83e436e3b0f52e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f4e07ba6584754b3d0e611cf3b9fa3" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 37049 } } peers { permanent_uuid: "f014460be902419ebb83e436e3b0f52e" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33705 } } peers { permanent_uuid: "7c4cedcfe9a64d9d83e9b721e5c83606" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 34389 } }
I20240507 02:11:08.110216 19513 leader_election.cc:288] T 1d9f865d67944d5e9786967f92c60607 P f014460be902419ebb83e436e3b0f52e [CANDIDATE]: Term 1 election: Requested vote from peers 10f4e07ba6584754b3d0e611cf3b9fa3 (127.31.131.67:37049), 7c4cedcfe9a64d9d83e9b721e5c83606 (127.31.131.66:34389)
W20240507 02:11:08.110250 19114 leader_election.cc:341] T 1d9f865d67944d5e9786967f92c60607 P f014460be902419ebb83e436e3b0f52e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10f4e07ba6584754b3d0e611cf3b9fa3 (127.31.131.67:37049): Illegal state: must be running to vote when last-logged opid is not known
I20240507 02:11:08.110486 19448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d9f865d67944d5e9786967f92c60607" candidate_uuid: "f014460be902419ebb83e436e3b0f52e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10f4e07ba6584754b3d0e611cf3b9fa3"
I20240507 02:11:08.110502 19314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d9f865d67944d5e9786967f92c60607" candidate_uuid: "f014460be902419ebb83e436e3b0f52e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c4cedcfe9a64d9d83e9b721e5c83606"
I20240507 02:11:08.110590 19314 raft_consensus.cc:3037] T 1d9f865d67944d5e9786967f92c60607 P 7c4cedcfe9a64d9d83e9b721e5c83606 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:11:08.110816 19514 raft_consensus.cc:348] T 1d9f865d67944d5e9786967f92c60607 P 10f4e07ba6584754b3d0e611cf3b9fa3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f4e07ba6584754b3d0e611cf3b9fa3" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 37049 } } peers { permanent_uuid: "f014460be902419ebb83e436e3b0f52e" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33705 } } peers { permanent_uuid: "7c4cedcfe9a64d9d83e9b721e5c83606" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 34389 } }
I20240507 02:11:08.110900 19514 raft_consensus.cc:374] T 1d9f865d67944d5e9786967f92c60607 P 10f4e07ba6584754b3d0e611cf3b9fa3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:11:08.110917 19514 raft_consensus.cc:729] T 1d9f865d67944d5e9786967f92c60607 P 10f4e07ba6584754b3d0e611cf3b9fa3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f4e07ba6584754b3d0e611cf3b9fa3, State: Initialized, Role: FOLLOWER
I20240507 02:11:08.110987 19514 consensus_queue.cc:260] T 1d9f865d67944d5e9786967f92c60607 P 10f4e07ba6584754b3d0e611cf3b9fa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f4e07ba6584754b3d0e611cf3b9fa3" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 37049 } } peers { permanent_uuid: "f014460be902419ebb83e436e3b0f52e" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33705 } } peers { permanent_uuid: "7c4cedcfe9a64d9d83e9b721e5c83606" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 34389 } }
I20240507 02:11:08.111161 19514 ts_tablet_manager.cc:1398] T 1d9f865d67944d5e9786967f92c60607 P 10f4e07ba6584754b3d0e611cf3b9fa3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240507 02:11:08.111222 19494 heartbeater.cc:503] Master 127.31.131.126:42727 was elected leader, sending a full tablet report...
I20240507 02:11:08.111281 19314 raft_consensus.cc:2454] T 1d9f865d67944d5e9786967f92c60607 P 7c4cedcfe9a64d9d83e9b721e5c83606 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f014460be902419ebb83e436e3b0f52e in term 1.
I20240507 02:11:08.111440 19115 leader_election.cc:302] T 1d9f865d67944d5e9786967f92c60607 P f014460be902419ebb83e436e3b0f52e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c4cedcfe9a64d9d83e9b721e5c83606, f014460be902419ebb83e436e3b0f52e; no voters:
W20240507 02:11:08.111522 19114 leader_election.cc:341] T 1d9f865d67944d5e9786967f92c60607 P f014460be902419ebb83e436e3b0f52e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 10f4e07ba6584754b3d0e611cf3b9fa3 (127.31.131.67:37049): Illegal state: must be running to vote when last-logged opid is not known
I20240507 02:11:08.111676 19513 raft_consensus.cc:2781] T 1d9f865d67944d5e9786967f92c60607 P f014460be902419ebb83e436e3b0f52e [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:11:08.111721 19513 raft_consensus.cc:686] T 1d9f865d67944d5e9786967f92c60607 P f014460be902419ebb83e436e3b0f52e [term 1 LEADER]: Becoming Leader. State: Replica: f014460be902419ebb83e436e3b0f52e, State: Running, Role: LEADER
I20240507 02:11:08.111785 19513 consensus_queue.cc:237] T 1d9f865d67944d5e9786967f92c60607 P f014460be902419ebb83e436e3b0f52e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f4e07ba6584754b3d0e611cf3b9fa3" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 37049 } } peers { permanent_uuid: "f014460be902419ebb83e436e3b0f52e" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33705 } } peers { permanent_uuid: "7c4cedcfe9a64d9d83e9b721e5c83606" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 34389 } }
I20240507 02:11:08.112360 19033 catalog_manager.cc:5484] T 1d9f865d67944d5e9786967f92c60607 P f014460be902419ebb83e436e3b0f52e reported cstate change: term changed from 0 to 1, leader changed from <none> to f014460be902419ebb83e436e3b0f52e (127.31.131.65). New cstate: current_term: 1 leader_uuid: "f014460be902419ebb83e436e3b0f52e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f4e07ba6584754b3d0e611cf3b9fa3" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 37049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f014460be902419ebb83e436e3b0f52e" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 33705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c4cedcfe9a64d9d83e9b721e5c83606" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 34389 } health_report { overall_health: UNKNOWN } } }
I20240507 02:11:08.119643 19505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:08.117975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45630 (local address 127.31.131.65:33705)
0507 02:11:08.118018 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:08.118020 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:08.118027 (+ 7us) server_negotiation.cc:407] Connection header received
0507 02:11:08.118050 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:08.118051 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:08.118062 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:08.118116 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0507 02:11:08.118298 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:08.118773 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:08.119285 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:08.119387 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:08.119413 (+ 26us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0507 02:11:08.119475 (+ 62us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0507 02:11:08.119537 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:08.119580 (+ 43us) server_negotiation.cc:299] Negotiation successful
0507 02:11:08.119594 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20240507 02:11:08.120851 19448 raft_consensus.cc:3037] T 1d9f865d67944d5e9786967f92c60607 P 10f4e07ba6584754b3d0e611cf3b9fa3 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:11:08.120878 19314 raft_consensus.cc:1261] T 1d9f865d67944d5e9786967f92c60607 P 7c4cedcfe9a64d9d83e9b721e5c83606 [term 1 FOLLOWER]: Refusing update from remote peer f014460be902419ebb83e436e3b0f52e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 02:11:08.121125 19512 consensus_queue.cc:1035] T 1d9f865d67944d5e9786967f92c60607 P f014460be902419ebb83e436e3b0f52e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c4cedcfe9a64d9d83e9b721e5c83606" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 34389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 02:11:08.121553 19448 raft_consensus.cc:1261] T 1d9f865d67944d5e9786967f92c60607 P 10f4e07ba6584754b3d0e611cf3b9fa3 [term 1 FOLLOWER]: Refusing update from remote peer f014460be902419ebb83e436e3b0f52e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 02:11:08.121783 19512 consensus_queue.cc:1035] T 1d9f865d67944d5e9786967f92c60607 P f014460be902419ebb83e436e3b0f52e [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f4e07ba6584754b3d0e611cf3b9fa3" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 37049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 02:11:08.122992 19521 mvcc.cc:205] Tried to move back new op lower bound from 7024836067821289472 to 7024836067786436608. Current Snapshot: MvccSnapshot[applied={T|T < 7024836067821289472}]
I20240507 02:11:08.124143 19524 mvcc.cc:205] Tried to move back new op lower bound from 7024836067821289472 to 7024836067786436608. Current Snapshot: MvccSnapshot[applied={T|T < 7024836067821289472 or (T in {7024836067821289472})}]
I20240507 02:11:08.125816 19033 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55236:
table { table_name: "test-table" } modify_external_catalogs: true
I20240507 02:11:08.125897 19033 catalog_manager.cc:2642] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55236:
table { table_name: "test-table" } modify_external_catalogs: true
I20240507 02:11:08.126487 19033 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e57407ea8a1c4fa78e1826b414d0fd3c: Sending DeleteTablet for 3 replicas of tablet 1d9f865d67944d5e9786967f92c60607
I20240507 02:11:08.126806 19428 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d9f865d67944d5e9786967f92c60607 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:11:08 UTC) from {username='kudu', principal='kudu/127.31.131.126@KRBTEST.COM'} at 127.0.0.1:53748
I20240507 02:11:08.126847 19159 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d9f865d67944d5e9786967f92c60607 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:11:08 UTC) from {username='kudu', principal='kudu/127.31.131.126@KRBTEST.COM'} at 127.0.0.1:45620
I20240507 02:11:08.127038 19530 tablet_replica.cc:330] T 1d9f865d67944d5e9786967f92c60607 P f014460be902419ebb83e436e3b0f52e: stopping tablet replica
I20240507 02:11:08.127032 19294 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d9f865d67944d5e9786967f92c60607 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:11:08 UTC) from {username='kudu', principal='kudu/127.31.131.126@KRBTEST.COM'} at 127.0.0.1:60506
I20240507 02:11:08.127117 19530 raft_consensus.cc:2229] T 1d9f865d67944d5e9786967f92c60607 P f014460be902419ebb83e436e3b0f52e [term 1 LEADER]: Raft consensus shutting down.
I20240507 02:11:08.127204 19530 raft_consensus.cc:2258] T 1d9f865d67944d5e9786967f92c60607 P f014460be902419ebb83e436e3b0f52e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 02:11:08.127372 19531 tablet_replica.cc:330] T 1d9f865d67944d5e9786967f92c60607 P 7c4cedcfe9a64d9d83e9b721e5c83606: stopping tablet replica
I20240507 02:11:08.127446 19531 raft_consensus.cc:2229] T 1d9f865d67944d5e9786967f92c60607 P 7c4cedcfe9a64d9d83e9b721e5c83606 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 02:11:08.127528 19531 raft_consensus.cc:2258] T 1d9f865d67944d5e9786967f92c60607 P 7c4cedcfe9a64d9d83e9b721e5c83606 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 02:11:08.127614 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 19096
I20240507 02:11:08.128777 19531 ts_tablet_manager.cc:1875] T 1d9f865d67944d5e9786967f92c60607 P 7c4cedcfe9a64d9d83e9b721e5c83606: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 02:11:08.129236 19532 tablet_replica.cc:330] T 1d9f865d67944d5e9786967f92c60607 P 10f4e07ba6584754b3d0e611cf3b9fa3: stopping tablet replica
I20240507 02:11:08.129312 19532 raft_consensus.cc:2229] T 1d9f865d67944d5e9786967f92c60607 P 10f4e07ba6584754b3d0e611cf3b9fa3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 02:11:08.129570 19532 raft_consensus.cc:2258] T 1d9f865d67944d5e9786967f92c60607 P 10f4e07ba6584754b3d0e611cf3b9fa3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 02:11:08.129956 19532 ts_tablet_manager.cc:1875] T 1d9f865d67944d5e9786967f92c60607 P 10f4e07ba6584754b3d0e611cf3b9fa3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 02:11:08.131255 19532 ts_tablet_manager.cc:1888] T 1d9f865d67944d5e9786967f92c60607 P 10f4e07ba6584754b3d0e611cf3b9fa3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240507 02:11:08.131325 19532 log.cc:1196] T 1d9f865d67944d5e9786967f92c60607 P 10f4e07ba6584754b3d0e611cf3b9fa3: Deleting WAL directory at /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/minicluster-data/ts-2/wal/wals/1d9f865d67944d5e9786967f92c60607
I20240507 02:11:08.131453 19531 ts_tablet_manager.cc:1888] T 1d9f865d67944d5e9786967f92c60607 P 7c4cedcfe9a64d9d83e9b721e5c83606: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240507 02:11:08.131515 19531 log.cc:1196] T 1d9f865d67944d5e9786967f92c60607 P 7c4cedcfe9a64d9d83e9b721e5c83606: Deleting WAL directory at /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1715047604443657-32269-0/minicluster-data/ts-1/wal/wals/1d9f865d67944d5e9786967f92c60607
I20240507 02:11:08.131608 19532 ts_tablet_manager.cc:1909] T 1d9f865d67944d5e9786967f92c60607 P 10f4e07ba6584754b3d0e611cf3b9fa3: Deleting consensus metadata
I20240507 02:11:08.131764 19531 ts_tablet_manager.cc:1909] T 1d9f865d67944d5e9786967f92c60607 P 7c4cedcfe9a64d9d83e9b721e5c83606: Deleting consensus metadata
I20240507 02:11:08.131955 19020 catalog_manager.cc:4830] TS 10f4e07ba6584754b3d0e611cf3b9fa3 (127.31.131.67:37049): tablet 1d9f865d67944d5e9786967f92c60607 (table test-table [id=e70927d60db646d6be1cfa3efc24eb73]) successfully deleted
I20240507 02:11:08.132122 19021 catalog_manager.cc:4830] TS 7c4cedcfe9a64d9d83e9b721e5c83606 (127.31.131.66:34389): tablet 1d9f865d67944d5e9786967f92c60607 (table test-table [id=e70927d60db646d6be1cfa3efc24eb73]) successfully deleted
W20240507 02:11:08.133468 19021 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.131.65:33705 (error 108)
I20240507 02:11:08.133721 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 19231
I20240507 02:11:08.133863 19503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:08.133758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.65:33705 (local address 127.0.0.1:45646)
0507 02:11:08.133807 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:08.133839 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.65:33705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20240507 02:11:08.135739 19021 catalog_manager.cc:4573] TS f014460be902419ebb83e436e3b0f52e (127.31.131.65:33705): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1d9f865d67944d5e9786967f92c60607: Network error: Client connection negotiation failed: client connection to 127.31.131.65:33705: connect: Connection refused (error 111)
I20240507 02:11:08.138252 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 19365
I20240507 02:11:08.142048 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 19002
2024-05-07T02:11:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1634 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2024-05-07T02:11:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-07T02:11:08Z Disabled control of system clock
I20240507 02:11:08.165632 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715047604443657-32269-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715047604443657-32269-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715047604443657-32269-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:46127
--webserver_interface=127.31.131.126
--webserver_port=0
--builtin_ntp_servers=127.31.131.84:43949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.131.126:46127
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20240507 02:11:08.239056 19537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:11:08.239207 19537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:11:08.239226 19537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:11:08.240458 19537 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240507 02:11:08.240536 19537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:11:08.240550 19537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:11:08.240643 19537 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240507 02:11:08.240685 19537 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240507 02:11:08.242116 19537 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:43949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715047604443657-32269-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715047604443657-32269-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.131.126:46127
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:46127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.131.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715047604443657-32269-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:11:08.242276 19537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:11:08.242475 19537 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:11:08.246464 19543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:08.246528 19537 server_base.cc:1034] running on GCE node
W20240507 02:11:08.246647 19545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:11:08.246471 19542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:08.246881 19537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:11:08.247090 19537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:11:08.248206 19537 hybrid_clock.cc:648] HybridClock initialized: now 1715047868248189 us; error 32 us; skew 500 ppm
I20240507 02:11:08.249151 19537 webserver.cc:460] Webserver started at http://127.31.131.126:46783/ using document root <none> and password file <none>
I20240507 02:11:08.249302 19537 fs_manager.cc:359] Metadata directory not provided
I20240507 02:11:08.249338 19537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:11:08.249442 19537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:11:08.250303 19537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715047604443657-32269-0/minicluster-data/master-0/data/instance:
uuid: "2ce07dd3e33849769ae9d17865f24695"
format_stamp: "Formatted at 2024-05-07 02:11:08 on dist-test-slave-v0bg"
server_key: "8af5ba2693df6bcdcedc031544e9c5cd"
server_key_iv: "4ab54172145c515f2f09efc6bc6a3e6d"
server_key_version: "encryptionkey@0"
I20240507 02:11:08.250591 19537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715047604443657-32269-0/minicluster-data/master-0/wal/instance:
uuid: "2ce07dd3e33849769ae9d17865f24695"
format_stamp: "Formatted at 2024-05-07 02:11:08 on dist-test-slave-v0bg"
server_key: "8af5ba2693df6bcdcedc031544e9c5cd"
server_key_iv: "4ab54172145c515f2f09efc6bc6a3e6d"
server_key_version: "encryptionkey@0"
I20240507 02:11:08.251544 19537 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240507 02:11:08.252102 19551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:08.252246 19537 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240507 02:11:08.252305 19537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715047604443657-32269-0/minicluster-data/master-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715047604443657-32269-0/minicluster-data/master-0/wal
uuid: "2ce07dd3e33849769ae9d17865f24695"
format_stamp: "Formatted at 2024-05-07 02:11:08 on dist-test-slave-v0bg"
server_key: "8af5ba2693df6bcdcedc031544e9c5cd"
server_key_iv: "4ab54172145c515f2f09efc6bc6a3e6d"
server_key_version: "encryptionkey@0"
I20240507 02:11:08.252362 19537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715047604443657-32269-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715047604443657-32269-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715047604443657-32269-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:11:08.278158 19537 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:11:08.278347 19537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:11:08.278422 19537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:11:08.281319 19537 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.126:46127
I20240507 02:11:08.281349 19603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.126:46127 every 8 connection(s)
I20240507 02:11:08.281592 19537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
I20240507 02:11:08.282086 19604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:11:08.284246 19604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ce07dd3e33849769ae9d17865f24695: Bootstrap starting.
I20240507 02:11:08.284736 19604 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ce07dd3e33849769ae9d17865f24695: Neither blocks nor log segments found. Creating new log.
I20240507 02:11:08.284955 19604 log.cc:824] T 00000000000000000000000000000000 P 2ce07dd3e33849769ae9d17865f24695: Log is configured to *not* fsync() on all Append() calls
I20240507 02:11:08.285503 19604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ce07dd3e33849769ae9d17865f24695: No bootstrap required, opened a new log
I20240507 02:11:08.286828 19604 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ce07dd3e33849769ae9d17865f24695 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ce07dd3e33849769ae9d17865f24695" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 46127 } }
I20240507 02:11:08.286933 19604 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ce07dd3e33849769ae9d17865f24695 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:11:08.286949 19604 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ce07dd3e33849769ae9d17865f24695 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ce07dd3e33849769ae9d17865f24695, State: Initialized, Role: FOLLOWER
I20240507 02:11:08.287060 19604 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ce07dd3e33849769ae9d17865f24695 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ce07dd3e33849769ae9d17865f24695" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 46127 } }
I20240507 02:11:08.287114 19604 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ce07dd3e33849769ae9d17865f24695 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 02:11:08.287134 19604 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ce07dd3e33849769ae9d17865f24695 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 02:11:08.287174 19604 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ce07dd3e33849769ae9d17865f24695 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:11:08.287652 19604 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ce07dd3e33849769ae9d17865f24695 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ce07dd3e33849769ae9d17865f24695" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 46127 } }
I20240507 02:11:08.287799 19604 leader_election.cc:302] T 00000000000000000000000000000000 P 2ce07dd3e33849769ae9d17865f24695 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ce07dd3e33849769ae9d17865f24695; no voters:
I20240507 02:11:08.287954 19604 leader_election.cc:288] T 00000000000000000000000000000000 P 2ce07dd3e33849769ae9d17865f24695 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240507 02:11:08.287997 19607 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ce07dd3e33849769ae9d17865f24695 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:11:08.288130 19607 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ce07dd3e33849769ae9d17865f24695 [term 1 LEADER]: Becoming Leader. State: Replica: 2ce07dd3e33849769ae9d17865f24695, State: Running, Role: LEADER
I20240507 02:11:08.288165 19604 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ce07dd3e33849769ae9d17865f24695 [sys.catalog]: configured and running, proceeding with master startup.
I20240507 02:11:08.288218 19607 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ce07dd3e33849769ae9d17865f24695 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ce07dd3e33849769ae9d17865f24695" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 46127 } }
I20240507 02:11:08.288529 19609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ce07dd3e33849769ae9d17865f24695 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ce07dd3e33849769ae9d17865f24695. Latest consensus state: current_term: 1 leader_uuid: "2ce07dd3e33849769ae9d17865f24695" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ce07dd3e33849769ae9d17865f24695" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 46127 } } }
I20240507 02:11:08.288620 19609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ce07dd3e33849769ae9d17865f24695 [sys.catalog]: This master's current role is: LEADER
I20240507 02:11:08.288826 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 19537
I20240507 02:11:08.288909 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715047604443657-32269-0/minicluster-data/master-0/wal/instance
I20240507 02:11:08.289044 32269 external_mini_cluster.cc:1428] Setting key a0df900cb9f541e7e4f6293f6ec3efe7
I20240507 02:11:08.289610 19614 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 02:11:08.288511 19608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ce07dd3e33849769ae9d17865f24695 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ce07dd3e33849769ae9d17865f24695" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ce07dd3e33849769ae9d17865f24695" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 46127 } } }
I20240507 02:11:08.289814 19608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ce07dd3e33849769ae9d17865f24695 [sys.catalog]: This master's current role is: LEADER
I20240507 02:11:08.290023 19614 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240507 02:11:08.291318 19614 catalog_manager.cc:1347] Generated new cluster ID: f5b33597e778428a8a853fda4dbf636e
I20240507 02:11:08.291368 19614 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240507 02:11:08.294073 19618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:08.289363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53838 (local address 127.31.131.126:46127)
0507 02:11:08.289958 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:08.289964 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:08.289983 (+ 19us) server_negotiation.cc:407] Connection header received
0507 02:11:08.290108 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:08.290115 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:08.290182 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:08.290934 (+ 752us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:08.291023 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:08.292137 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:08.292914 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:08.293421 (+ 507us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:08.293517 (+ 96us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:08.293535 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:08.293536 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0507 02:11:08.293703 (+ 167us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:08.293727 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0507 02:11:08.293733 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:08.293792 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:08.293871 (+ 79us) server_negotiation.cc:299] Negotiation successful
0507 02:11:08.293914 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":288,"threads_started":1}
I20240507 02:11:08.299988 19614 catalog_manager.cc:1370] Generated new certificate authority record
I20240507 02:11:08.300408 19614 catalog_manager.cc:1504] Loading token signing keys...
I20240507 02:11:08.304774 19614 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 2ce07dd3e33849769ae9d17865f24695: Generated new TSK 0
I20240507 02:11:08.304899 19614 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240507 02:11:08.309913 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715047604443657-32269-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715047604443657-32269-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715047604443657-32269-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.65:0
--local_ip_for_outbound_sockets=127.31.131.65
--webserver_interface=127.31.131.65
--webserver_port=0
--tserver_master_addrs=127.31.131.126:46127
--builtin_ntp_servers=127.31.131.84:43949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20240507 02:11:08.383158 19628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:11:08.383316 19628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:11:08.383337 19628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:11:08.384641 19628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:11:08.384683 19628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:11:08.384707 19628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.65
I20240507 02:11:08.386058 19628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:43949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715047604443657-32269-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715047604443657-32269-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.131.65
--webserver_port=0
--tserver_master_addrs=127.31.131.126:46127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.65
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715047604443657-32269-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:11:08.386238 19628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:11:08.386471 19628 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:11:08.390197 19633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:11:08.390205 19634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:11:08.390446 19636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:08.390697 19628 server_base.cc:1034] running on GCE node
I20240507 02:11:08.390832 19628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:11:08.391019 19628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:11:08.392145 19628 hybrid_clock.cc:648] HybridClock initialized: now 1715047868392135 us; error 25 us; skew 500 ppm
I20240507 02:11:08.393249 19628 webserver.cc:460] Webserver started at http://127.31.131.65:37279/ using document root <none> and password file <none>
I20240507 02:11:08.393419 19628 fs_manager.cc:359] Metadata directory not provided
I20240507 02:11:08.393453 19628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:11:08.393559 19628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:11:08.394454 19628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715047604443657-32269-0/minicluster-data/ts-0/data/instance:
uuid: "85d58e2824624d8ba53ce817aafee478"
format_stamp: "Formatted at 2024-05-07 02:11:08 on dist-test-slave-v0bg"
server_key: "66f1634d2e1d7afb368a2445a93f974a"
server_key_iv: "b4186eee51adaa743483878a3e346739"
server_key_version: "encryptionkey@0"
I20240507 02:11:08.394755 19628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715047604443657-32269-0/minicluster-data/ts-0/wal/instance:
uuid: "85d58e2824624d8ba53ce817aafee478"
format_stamp: "Formatted at 2024-05-07 02:11:08 on dist-test-slave-v0bg"
server_key: "66f1634d2e1d7afb368a2445a93f974a"
server_key_iv: "b4186eee51adaa743483878a3e346739"
server_key_version: "encryptionkey@0"
I20240507 02:11:08.395757 19628 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240507 02:11:08.396330 19642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:08.396474 19628 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240507 02:11:08.396536 19628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715047604443657-32269-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715047604443657-32269-0/minicluster-data/ts-0/wal
uuid: "85d58e2824624d8ba53ce817aafee478"
format_stamp: "Formatted at 2024-05-07 02:11:08 on dist-test-slave-v0bg"
server_key: "66f1634d2e1d7afb368a2445a93f974a"
server_key_iv: "b4186eee51adaa743483878a3e346739"
server_key_version: "encryptionkey@0"
I20240507 02:11:08.396595 19628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715047604443657-32269-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715047604443657-32269-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715047604443657-32269-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:11:08.411232 19628 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:11:08.411407 19628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:11:08.411506 19628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:11:08.411661 19628 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:11:08.411942 19628 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:11:08.411975 19628 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:08.411993 19628 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:11:08.412004 19628 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:08.417869 19628 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.65:34585
I20240507 02:11:08.417923 19755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.65:34585 every 8 connection(s)
I20240507 02:11:08.418165 19628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715047604443657-32269-0/minicluster-data/ts-0/data/info.pb
I20240507 02:11:08.422765 19618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:08.418874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:49673 (local address 127.31.131.126:46127)
0507 02:11:08.418933 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:08.418935 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:08.419092 (+ 157us) server_negotiation.cc:407] Connection header received
0507 02:11:08.419663 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:08.419664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:08.419673 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:08.419712 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:08.420238 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:08.420700 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:08.421417 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:08.421524 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:08.422500 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:08.422503 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:08.422504 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0507 02:11:08.422507 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:08.422521 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0507 02:11:08.422524 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:08.422596 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:08.422679 (+ 83us) server_negotiation.cc:299] Negotiation successful
0507 02:11:08.422709 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20240507 02:11:08.422891 19758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:08.418810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:46127 (local address 127.31.131.65:49673)
0507 02:11:08.418994 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:08.419024 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0507 02:11:08.419563 (+ 539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:11:08.419765 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:11:08.419769 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:11:08.419779 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:11:08.420166 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:08.420170 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:08.420779 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:11:08.420780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:11:08.421344 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:08.421346 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:08.421487 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:08.422370 (+ 883us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0507 02:11:08.422371 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:11:08.422393 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0507 02:11:08.422413 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:11:08.422598 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:11:08.422602 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:11:08.422604 (+ 2us) client_negotiation.cc:769] Sending connection context
0507 02:11:08.422676 (+ 72us) client_negotiation.cc:240] Negotiation successful
0507 02:11:08.422739 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":70,"threads_started":1}
I20240507 02:11:08.423380 19756 heartbeater.cc:347] Connected to a master server at 127.31.131.126:46127
I20240507 02:11:08.423462 19756 heartbeater.cc:464] Registering TS with master...
I20240507 02:11:08.423543 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 19628
I20240507 02:11:08.423631 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715047604443657-32269-0/minicluster-data/ts-0/wal/instance
I20240507 02:11:08.423672 19756 heartbeater.cc:511] Master 127.31.131.126:46127 requested a full tablet report, sending...
I20240507 02:11:08.423789 32269 external_mini_cluster.cc:1428] Setting key 4cdb4967043750d11ca00e6f8315bd60
I20240507 02:11:08.424237 19568 ts_manager.cc:194] Registered new tserver with Master: 85d58e2824624d8ba53ce817aafee478 (127.31.131.65:34585)
I20240507 02:11:08.424887 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715047604443657-32269-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715047604443657-32269-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715047604443657-32269-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.66:0
--local_ip_for_outbound_sockets=127.31.131.66
--webserver_interface=127.31.131.66
--webserver_port=0
--tserver_master_addrs=127.31.131.126:46127
--builtin_ntp_servers=127.31.131.84:43949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20240507 02:11:08.425151 19568 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.131.65:49673
W20240507 02:11:08.498775 19759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:11:08.498930 19759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:11:08.498957 19759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:11:08.500221 19759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:11:08.500255 19759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:11:08.500286 19759 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.66
I20240507 02:11:08.501626 19759 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:43949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715047604443657-32269-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715047604443657-32269-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.131.66
--webserver_port=0
--tserver_master_addrs=127.31.131.126:46127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.66
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715047604443657-32269-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:11:08.501816 19759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:11:08.502029 19759 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:11:08.505640 19765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:11:08.505652 19767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:08.505816 19759 server_base.cc:1034] running on GCE node
W20240507 02:11:08.505954 19764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:08.506127 19759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:11:08.506309 19759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:11:08.507426 19759 hybrid_clock.cc:648] HybridClock initialized: now 1715047868507413 us; error 25 us; skew 500 ppm
I20240507 02:11:08.508466 19759 webserver.cc:460] Webserver started at http://127.31.131.66:43605/ using document root <none> and password file <none>
I20240507 02:11:08.508630 19759 fs_manager.cc:359] Metadata directory not provided
I20240507 02:11:08.508668 19759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:11:08.508765 19759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:11:08.509619 19759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715047604443657-32269-0/minicluster-data/ts-1/data/instance:
uuid: "9bd96d22929e489f8d8ea8dfeadbe044"
format_stamp: "Formatted at 2024-05-07 02:11:08 on dist-test-slave-v0bg"
server_key: "468805859cb55654711a3bfb204a26fe"
server_key_iv: "e83896f51bd4b5e01f8c28416ea057a3"
server_key_version: "encryptionkey@0"
I20240507 02:11:08.509884 19759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715047604443657-32269-0/minicluster-data/ts-1/wal/instance:
uuid: "9bd96d22929e489f8d8ea8dfeadbe044"
format_stamp: "Formatted at 2024-05-07 02:11:08 on dist-test-slave-v0bg"
server_key: "468805859cb55654711a3bfb204a26fe"
server_key_iv: "e83896f51bd4b5e01f8c28416ea057a3"
server_key_version: "encryptionkey@0"
I20240507 02:11:08.510915 19759 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240507 02:11:08.511476 19773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:08.511626 19759 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240507 02:11:08.511695 19759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715047604443657-32269-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715047604443657-32269-0/minicluster-data/ts-1/wal
uuid: "9bd96d22929e489f8d8ea8dfeadbe044"
format_stamp: "Formatted at 2024-05-07 02:11:08 on dist-test-slave-v0bg"
server_key: "468805859cb55654711a3bfb204a26fe"
server_key_iv: "e83896f51bd4b5e01f8c28416ea057a3"
server_key_version: "encryptionkey@0"
I20240507 02:11:08.511749 19759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715047604443657-32269-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715047604443657-32269-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715047604443657-32269-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:11:08.543226 19759 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:11:08.543424 19759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:11:08.543526 19759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:11:08.543686 19759 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:11:08.543949 19759 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:11:08.543980 19759 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:08.544006 19759 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:11:08.544034 19759 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:08.548698 19759 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.66:45981
I20240507 02:11:08.548758 19886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.66:45981 every 8 connection(s)
I20240507 02:11:08.548979 19759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715047604443657-32269-0/minicluster-data/ts-1/data/info.pb
I20240507 02:11:08.552872 19618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:08.549534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.66:49275 (local address 127.31.131.126:46127)
0507 02:11:08.549573 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:08.549575 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:08.549758 (+ 183us) server_negotiation.cc:407] Connection header received
0507 02:11:08.550184 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:08.550185 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:08.550197 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:08.550246 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:08.550719 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:08.551196 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:08.551694 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:08.551855 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:08.552626 (+ 771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:08.552629 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:08.552630 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0507 02:11:08.552634 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:08.552648 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0507 02:11:08.552651 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:08.552693 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:08.552765 (+ 72us) server_negotiation.cc:299] Negotiation successful
0507 02:11:08.552798 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20240507 02:11:08.552968 19889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:08.549511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:46127 (local address 127.31.131.66:49275)
0507 02:11:08.549673 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:08.549693 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0507 02:11:08.550096 (+ 403us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:11:08.550304 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:11:08.550307 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:11:08.550316 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:11:08.550635 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:08.550638 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:08.551274 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:11:08.551275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:11:08.551626 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:08.551627 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:08.551737 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:08.552513 (+ 776us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0507 02:11:08.552513 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:11:08.552530 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0507 02:11:08.552546 (+ 16us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:11:08.552701 (+ 155us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:11:08.552705 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:11:08.552707 (+ 2us) client_negotiation.cc:769] Sending connection context
0507 02:11:08.552766 (+ 59us) client_negotiation.cc:240] Negotiation successful
0507 02:11:08.552824 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":78,"threads_started":1}
I20240507 02:11:08.553371 19887 heartbeater.cc:347] Connected to a master server at 127.31.131.126:46127
I20240507 02:11:08.553444 19887 heartbeater.cc:464] Registering TS with master...
I20240507 02:11:08.553588 19887 heartbeater.cc:511] Master 127.31.131.126:46127 requested a full tablet report, sending...
I20240507 02:11:08.553889 19568 ts_manager.cc:194] Registered new tserver with Master: 9bd96d22929e489f8d8ea8dfeadbe044 (127.31.131.66:45981)
I20240507 02:11:08.554212 19568 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.131.66:49275
I20240507 02:11:08.558787 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 19759
I20240507 02:11:08.558872 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715047604443657-32269-0/minicluster-data/ts-1/wal/instance
I20240507 02:11:08.559020 32269 external_mini_cluster.cc:1428] Setting key 6ca22fafb69f7c7e5b3011d10a600cd4
I20240507 02:11:08.559728 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715047604443657-32269-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715047604443657-32269-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715047604443657-32269-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.131.67:0
--local_ip_for_outbound_sockets=127.31.131.67
--webserver_interface=127.31.131.67
--webserver_port=0
--tserver_master_addrs=127.31.131.126:46127
--builtin_ntp_servers=127.31.131.84:43949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20240507 02:11:08.632622 19890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:11:08.632786 19890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:11:08.632809 19890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:11:08.634001 19890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:11:08.634039 19890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:11:08.634061 19890 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.131.67
I20240507 02:11:08.635401 19890 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:43949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715047604443657-32269-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715047604443657-32269-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.131.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.131.67
--webserver_port=0
--tserver_master_addrs=127.31.131.126:46127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.131.67
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715047604443657-32269-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:11:08.635547 19890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:11:08.635722 19890 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:11:08.639140 19896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:08.639173 19890 server_base.cc:1034] running on GCE node
W20240507 02:11:08.639153 19898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:11:08.639309 19895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:08.639510 19890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:11:08.639698 19890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:11:08.640820 19890 hybrid_clock.cc:648] HybridClock initialized: now 1715047868640809 us; error 29 us; skew 500 ppm
I20240507 02:11:08.641793 19890 webserver.cc:460] Webserver started at http://127.31.131.67:36149/ using document root <none> and password file <none>
I20240507 02:11:08.641940 19890 fs_manager.cc:359] Metadata directory not provided
I20240507 02:11:08.641978 19890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:11:08.642076 19890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:11:08.642953 19890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715047604443657-32269-0/minicluster-data/ts-2/data/instance:
uuid: "7df60f1707b24a9092cf97422009f21e"
format_stamp: "Formatted at 2024-05-07 02:11:08 on dist-test-slave-v0bg"
server_key: "c104f2b57a36dc4f5f4db4b5cf52b170"
server_key_iv: "4af7133f0fc399a6df6b34fb8405ba6c"
server_key_version: "encryptionkey@0"
I20240507 02:11:08.643215 19890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715047604443657-32269-0/minicluster-data/ts-2/wal/instance:
uuid: "7df60f1707b24a9092cf97422009f21e"
format_stamp: "Formatted at 2024-05-07 02:11:08 on dist-test-slave-v0bg"
server_key: "c104f2b57a36dc4f5f4db4b5cf52b170"
server_key_iv: "4af7133f0fc399a6df6b34fb8405ba6c"
server_key_version: "encryptionkey@0"
I20240507 02:11:08.644176 19890 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240507 02:11:08.644712 19904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:08.644850 19890 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240507 02:11:08.644913 19890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715047604443657-32269-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715047604443657-32269-0/minicluster-data/ts-2/wal
uuid: "7df60f1707b24a9092cf97422009f21e"
format_stamp: "Formatted at 2024-05-07 02:11:08 on dist-test-slave-v0bg"
server_key: "c104f2b57a36dc4f5f4db4b5cf52b170"
server_key_iv: "4af7133f0fc399a6df6b34fb8405ba6c"
server_key_version: "encryptionkey@0"
I20240507 02:11:08.644964 19890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715047604443657-32269-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715047604443657-32269-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715047604443657-32269-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:11:08.650787 19890 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:11:08.650946 19890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:11:08.651037 19890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:11:08.651180 19890 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 02:11:08.651402 19890 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 02:11:08.651432 19890 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:08.651454 19890 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 02:11:08.651480 19890 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:08.656158 19890 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.67:43123
I20240507 02:11:08.656195 20017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.67:43123 every 8 connection(s)
I20240507 02:11:08.656404 19890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715047604443657-32269-0/minicluster-data/ts-2/data/info.pb
I20240507 02:11:08.659878 19618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:08.656930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.67:38469 (local address 127.31.131.126:46127)
0507 02:11:08.656974 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:08.656976 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:08.657143 (+ 167us) server_negotiation.cc:407] Connection header received
0507 02:11:08.657498 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:08.657499 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:08.657511 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:08.657559 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:08.657928 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:08.658387 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:08.658862 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:08.658965 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:08.659650 (+ 685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:08.659653 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:08.659654 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0507 02:11:08.659657 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:08.659675 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0507 02:11:08.659678 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:08.659717 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:08.659778 (+ 61us) server_negotiation.cc:299] Negotiation successful
0507 02:11:08.659807 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20240507 02:11:08.659945 20020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:08.656908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:46127 (local address 127.31.131.67:38469)
0507 02:11:08.657061 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:08.657085 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0507 02:11:08.657433 (+ 348us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:11:08.657605 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:11:08.657608 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:11:08.657616 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:11:08.657873 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:08.657876 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:08.658441 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:11:08.658442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:11:08.658812 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:08.658814 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:08.658899 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:08.659555 (+ 656us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0507 02:11:08.659555 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:11:08.659569 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0507 02:11:08.659583 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:11:08.659726 (+ 143us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:11:08.659728 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:11:08.659729 (+ 1us) client_negotiation.cc:769] Sending connection context
0507 02:11:08.659775 (+ 46us) client_negotiation.cc:240] Negotiation successful
0507 02:11:08.659833 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":55,"threads_started":1}
I20240507 02:11:08.660305 20018 heartbeater.cc:347] Connected to a master server at 127.31.131.126:46127
I20240507 02:11:08.660378 20018 heartbeater.cc:464] Registering TS with master...
I20240507 02:11:08.660534 20018 heartbeater.cc:511] Master 127.31.131.126:46127 requested a full tablet report, sending...
I20240507 02:11:08.660830 19568 ts_manager.cc:194] Registered new tserver with Master: 7df60f1707b24a9092cf97422009f21e (127.31.131.67:43123)
I20240507 02:11:08.661175 19568 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.131.67:38469
I20240507 02:11:08.663146 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 19890
I20240507 02:11:08.663229 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715047604443657-32269-0/minicluster-data/ts-2/wal/instance
I20240507 02:11:08.663354 32269 external_mini_cluster.cc:1428] Setting key eb2ed89f501cf66575679e9fe5789b5a
I20240507 02:11:08.664043 32269 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240507 02:11:08.667300 19618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:08.665181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53848 (local address 127.31.131.126:46127)
0507 02:11:08.665225 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:08.665227 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:08.665318 (+ 91us) server_negotiation.cc:407] Connection header received
0507 02:11:08.665376 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:08.665377 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:08.665385 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:08.665428 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:08.665699 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:08.666119 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:08.666668 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:08.666817 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:08.667100 (+ 283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:08.667103 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:08.667103 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0507 02:11:08.667107 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:08.667118 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0507 02:11:08.667121 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:08.667162 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:08.667212 (+ 50us) server_negotiation.cc:299] Negotiation successful
0507 02:11:08.667235 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20240507 02:11:08.668512 19568 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53848:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 02:11:08.669791 19568 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 02:11:08.673885 20026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:08.671495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.67:43123 (local address 127.0.0.1:33838)
0507 02:11:08.671749 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:08.671772 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0507 02:11:08.671818 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:11:08.672035 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:11:08.672037 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:11:08.672039 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:11:08.672179 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:08.672180 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:08.673120 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:11:08.673121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:11:08.673654 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:08.673656 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:08.673710 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:08.673743 (+ 33us) client_negotiation.cc:769] Sending connection context
0507 02:11:08.673786 (+ 43us) client_negotiation.cc:240] Negotiation successful
0507 02:11:08.673821 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":40,"threads_started":1}
I20240507 02:11:08.674732 20028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:08.671622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33838 (local address 127.31.131.67:43123)
0507 02:11:08.671921 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:08.671926 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:08.671934 (+ 8us) server_negotiation.cc:407] Connection header received
0507 02:11:08.671954 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:08.671956 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:08.671965 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:08.672030 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:11:08.672251 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:08.673051 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:08.674009 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:08.674510 (+ 501us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:08.674604 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:08.674630 (+ 26us) server_negotiation.cc:299] Negotiation successful
0507 02:11:08.674662 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":67,"threads_started":1}
I20240507 02:11:08.674782 20027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:08.671533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.65:34585 (local address 127.0.0.1:44740)
0507 02:11:08.671778 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:08.671788 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0507 02:11:08.671843 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:11:08.672228 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:11:08.672229 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:11:08.672231 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:11:08.672442 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:08.672444 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:08.673724 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:11:08.673725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:11:08.674595 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:08.674598 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:08.674661 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:08.674664 (+ 3us) client_negotiation.cc:769] Sending connection context
0507 02:11:08.674691 (+ 27us) client_negotiation.cc:240] Negotiation successful
0507 02:11:08.674726 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":73,"threads_started":1}
I20240507 02:11:08.674934 20030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:08.671667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.66:45981 (local address 127.0.0.1:38826)
0507 02:11:08.672306 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:08.672315 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0507 02:11:08.672363 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:11:08.672597 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:11:08.672598 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:11:08.672601 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:11:08.672785 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:08.672787 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:08.673942 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:11:08.673943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:11:08.674745 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:08.674749 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:08.674800 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:08.674803 (+ 3us) client_negotiation.cc:769] Sending connection context
0507 02:11:08.674833 (+ 30us) client_negotiation.cc:240] Negotiation successful
0507 02:11:08.674871 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":52,"threads_started":1}
I20240507 02:11:08.675173 19952 tablet_service.cc:1467] Processing CreateTablet for tablet 3034413da5cc4bb4bebb66e17b7edbcc (DEFAULT_TABLE table=test-table [id=fce06915c1804713a689e217f8378380]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:11:08.675418 19952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3034413da5cc4bb4bebb66e17b7edbcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:11:08.675534 20029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:08.671634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44740 (local address 127.31.131.65:34585)
0507 02:11:08.672108 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:08.672113 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:08.672119 (+ 6us) server_negotiation.cc:407] Connection header received
0507 02:11:08.672145 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:08.672146 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:08.672156 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:08.672212 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:11:08.672519 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:08.673641 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:08.674859 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:08.675338 (+ 479us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:08.675405 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:08.675434 (+ 29us) server_negotiation.cc:299] Negotiation successful
0507 02:11:08.675468 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":76,"threads_started":1}
I20240507 02:11:08.675825 20031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:08.671762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38826 (local address 127.31.131.66:45981)
0507 02:11:08.671930 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:08.671936 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:08.672408 (+ 472us) server_negotiation.cc:407] Connection header received
0507 02:11:08.672445 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:08.672447 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:08.672462 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:08.672525 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:11:08.672855 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:08.673873 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:08.675039 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:08.675429 (+ 390us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:08.675488 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:08.675518 (+ 30us) server_negotiation.cc:299] Negotiation successful
0507 02:11:08.675557 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":80,"threads_started":1}
I20240507 02:11:08.676095 19690 tablet_service.cc:1467] Processing CreateTablet for tablet 3034413da5cc4bb4bebb66e17b7edbcc (DEFAULT_TABLE table=test-table [id=fce06915c1804713a689e217f8378380]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:11:08.676225 19821 tablet_service.cc:1467] Processing CreateTablet for tablet 3034413da5cc4bb4bebb66e17b7edbcc (DEFAULT_TABLE table=test-table [id=fce06915c1804713a689e217f8378380]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 02:11:08.676367 19690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3034413da5cc4bb4bebb66e17b7edbcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:11:08.676468 19821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3034413da5cc4bb4bebb66e17b7edbcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:11:08.678280 20032 tablet_bootstrap.cc:492] T 3034413da5cc4bb4bebb66e17b7edbcc P 7df60f1707b24a9092cf97422009f21e: Bootstrap starting.
I20240507 02:11:08.678539 20033 tablet_bootstrap.cc:492] T 3034413da5cc4bb4bebb66e17b7edbcc P 9bd96d22929e489f8d8ea8dfeadbe044: Bootstrap starting.
I20240507 02:11:08.678828 20032 tablet_bootstrap.cc:654] T 3034413da5cc4bb4bebb66e17b7edbcc P 7df60f1707b24a9092cf97422009f21e: Neither blocks nor log segments found. Creating new log.
I20240507 02:11:08.679069 20032 log.cc:824] T 3034413da5cc4bb4bebb66e17b7edbcc P 7df60f1707b24a9092cf97422009f21e: Log is configured to *not* fsync() on all Append() calls
I20240507 02:11:08.679095 20033 tablet_bootstrap.cc:654] T 3034413da5cc4bb4bebb66e17b7edbcc P 9bd96d22929e489f8d8ea8dfeadbe044: Neither blocks nor log segments found. Creating new log.
I20240507 02:11:08.679286 20034 tablet_bootstrap.cc:492] T 3034413da5cc4bb4bebb66e17b7edbcc P 85d58e2824624d8ba53ce817aafee478: Bootstrap starting.
I20240507 02:11:08.679373 20033 log.cc:824] T 3034413da5cc4bb4bebb66e17b7edbcc P 9bd96d22929e489f8d8ea8dfeadbe044: Log is configured to *not* fsync() on all Append() calls
I20240507 02:11:08.679811 20034 tablet_bootstrap.cc:654] T 3034413da5cc4bb4bebb66e17b7edbcc P 85d58e2824624d8ba53ce817aafee478: Neither blocks nor log segments found. Creating new log.
I20240507 02:11:08.679816 20032 tablet_bootstrap.cc:492] T 3034413da5cc4bb4bebb66e17b7edbcc P 7df60f1707b24a9092cf97422009f21e: No bootstrap required, opened a new log
I20240507 02:11:08.679872 20032 ts_tablet_manager.cc:1367] T 3034413da5cc4bb4bebb66e17b7edbcc P 7df60f1707b24a9092cf97422009f21e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240507 02:11:08.680030 20034 log.cc:824] T 3034413da5cc4bb4bebb66e17b7edbcc P 85d58e2824624d8ba53ce817aafee478: Log is configured to *not* fsync() on all Append() calls
I20240507 02:11:08.680038 20033 tablet_bootstrap.cc:492] T 3034413da5cc4bb4bebb66e17b7edbcc P 9bd96d22929e489f8d8ea8dfeadbe044: No bootstrap required, opened a new log
I20240507 02:11:08.680089 20033 ts_tablet_manager.cc:1367] T 3034413da5cc4bb4bebb66e17b7edbcc P 9bd96d22929e489f8d8ea8dfeadbe044: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240507 02:11:08.680523 20034 tablet_bootstrap.cc:492] T 3034413da5cc4bb4bebb66e17b7edbcc P 85d58e2824624d8ba53ce817aafee478: No bootstrap required, opened a new log
I20240507 02:11:08.680562 20034 ts_tablet_manager.cc:1367] T 3034413da5cc4bb4bebb66e17b7edbcc P 85d58e2824624d8ba53ce817aafee478: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20240507 02:11:08.682005 20032 raft_consensus.cc:348] T 3034413da5cc4bb4bebb66e17b7edbcc P 7df60f1707b24a9092cf97422009f21e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df60f1707b24a9092cf97422009f21e" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 43123 } } peers { permanent_uuid: "9bd96d22929e489f8d8ea8dfeadbe044" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 45981 } } peers { permanent_uuid: "85d58e2824624d8ba53ce817aafee478" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 34585 } }
I20240507 02:11:08.682129 20032 raft_consensus.cc:374] T 3034413da5cc4bb4bebb66e17b7edbcc P 7df60f1707b24a9092cf97422009f21e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:11:08.682155 20032 raft_consensus.cc:729] T 3034413da5cc4bb4bebb66e17b7edbcc P 7df60f1707b24a9092cf97422009f21e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7df60f1707b24a9092cf97422009f21e, State: Initialized, Role: FOLLOWER
I20240507 02:11:08.682228 20034 raft_consensus.cc:348] T 3034413da5cc4bb4bebb66e17b7edbcc P 85d58e2824624d8ba53ce817aafee478 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df60f1707b24a9092cf97422009f21e" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 43123 } } peers { permanent_uuid: "9bd96d22929e489f8d8ea8dfeadbe044" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 45981 } } peers { permanent_uuid: "85d58e2824624d8ba53ce817aafee478" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 34585 } }
I20240507 02:11:08.682250 20032 consensus_queue.cc:260] T 3034413da5cc4bb4bebb66e17b7edbcc P 7df60f1707b24a9092cf97422009f21e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df60f1707b24a9092cf97422009f21e" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 43123 } } peers { permanent_uuid: "9bd96d22929e489f8d8ea8dfeadbe044" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 45981 } } peers { permanent_uuid: "85d58e2824624d8ba53ce817aafee478" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 34585 } }
I20240507 02:11:08.682313 20034 raft_consensus.cc:374] T 3034413da5cc4bb4bebb66e17b7edbcc P 85d58e2824624d8ba53ce817aafee478 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:11:08.682188 20033 raft_consensus.cc:348] T 3034413da5cc4bb4bebb66e17b7edbcc P 9bd96d22929e489f8d8ea8dfeadbe044 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df60f1707b24a9092cf97422009f21e" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 43123 } } peers { permanent_uuid: "9bd96d22929e489f8d8ea8dfeadbe044" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 45981 } } peers { permanent_uuid: "85d58e2824624d8ba53ce817aafee478" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 34585 } }
I20240507 02:11:08.682328 20034 raft_consensus.cc:729] T 3034413da5cc4bb4bebb66e17b7edbcc P 85d58e2824624d8ba53ce817aafee478 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85d58e2824624d8ba53ce817aafee478, State: Initialized, Role: FOLLOWER
I20240507 02:11:08.682336 20033 raft_consensus.cc:374] T 3034413da5cc4bb4bebb66e17b7edbcc P 9bd96d22929e489f8d8ea8dfeadbe044 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:11:08.682365 20033 raft_consensus.cc:729] T 3034413da5cc4bb4bebb66e17b7edbcc P 9bd96d22929e489f8d8ea8dfeadbe044 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bd96d22929e489f8d8ea8dfeadbe044, State: Initialized, Role: FOLLOWER
I20240507 02:11:08.682395 20034 consensus_queue.cc:260] T 3034413da5cc4bb4bebb66e17b7edbcc P 85d58e2824624d8ba53ce817aafee478 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df60f1707b24a9092cf97422009f21e" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 43123 } } peers { permanent_uuid: "9bd96d22929e489f8d8ea8dfeadbe044" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 45981 } } peers { permanent_uuid: "85d58e2824624d8ba53ce817aafee478" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 34585 } }
I20240507 02:11:08.682472 20032 ts_tablet_manager.cc:1398] T 3034413da5cc4bb4bebb66e17b7edbcc P 7df60f1707b24a9092cf97422009f21e: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240507 02:11:08.682487 20033 consensus_queue.cc:260] T 3034413da5cc4bb4bebb66e17b7edbcc P 9bd96d22929e489f8d8ea8dfeadbe044 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df60f1707b24a9092cf97422009f21e" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 43123 } } peers { permanent_uuid: "9bd96d22929e489f8d8ea8dfeadbe044" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 45981 } } peers { permanent_uuid: "85d58e2824624d8ba53ce817aafee478" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 34585 } }
I20240507 02:11:08.682596 20018 heartbeater.cc:503] Master 127.31.131.126:46127 was elected leader, sending a full tablet report...
I20240507 02:11:08.682663 19756 heartbeater.cc:503] Master 127.31.131.126:46127 was elected leader, sending a full tablet report...
I20240507 02:11:08.682874 20034 ts_tablet_manager.cc:1398] T 3034413da5cc4bb4bebb66e17b7edbcc P 85d58e2824624d8ba53ce817aafee478: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240507 02:11:08.683408 19887 heartbeater.cc:503] Master 127.31.131.126:46127 was elected leader, sending a full tablet report...
I20240507 02:11:08.683341 20033 ts_tablet_manager.cc:1398] T 3034413da5cc4bb4bebb66e17b7edbcc P 9bd96d22929e489f8d8ea8dfeadbe044: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
W20240507 02:11:08.799399 19888 tablet.cc:2369] T 3034413da5cc4bb4bebb66e17b7edbcc P 9bd96d22929e489f8d8ea8dfeadbe044: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240507 02:11:08.884085 20039 raft_consensus.cc:482] T 3034413da5cc4bb4bebb66e17b7edbcc P 85d58e2824624d8ba53ce817aafee478 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 02:11:08.884208 20039 raft_consensus.cc:504] T 3034413da5cc4bb4bebb66e17b7edbcc P 85d58e2824624d8ba53ce817aafee478 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df60f1707b24a9092cf97422009f21e" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 43123 } } peers { permanent_uuid: "9bd96d22929e489f8d8ea8dfeadbe044" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 45981 } } peers { permanent_uuid: "85d58e2824624d8ba53ce817aafee478" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 34585 } }
I20240507 02:11:08.884490 20039 leader_election.cc:288] T 3034413da5cc4bb4bebb66e17b7edbcc P 85d58e2824624d8ba53ce817aafee478 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7df60f1707b24a9092cf97422009f21e (127.31.131.67:43123), 9bd96d22929e489f8d8ea8dfeadbe044 (127.31.131.66:45981)
I20240507 02:11:08.886616 19758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:08.884639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.67:43123 (local address 127.31.131.65:44571)
0507 02:11:08.884759 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:08.884771 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0507 02:11:08.884858 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:11:08.885001 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:11:08.885002 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:11:08.885003 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:11:08.885173 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:08.885175 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:08.885831 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:11:08.885833 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:11:08.886374 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:08.886376 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:08.886443 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:08.886447 (+ 4us) client_negotiation.cc:769] Sending connection context
0507 02:11:08.886483 (+ 36us) client_negotiation.cc:240] Negotiation successful
0507 02:11:08.886521 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96}
I20240507 02:11:08.886732 20041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:08.884737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.66:45981 (local address 127.31.131.65:53725)
0507 02:11:08.884902 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:08.884910 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0507 02:11:08.884964 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:11:08.885087 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:11:08.885088 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:11:08.885089 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0507 02:11:08.885289 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:08.885290 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:08.885877 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:11:08.885879 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:11:08.886428 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:08.886430 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:08.886504 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:08.886507 (+ 3us) client_negotiation.cc:769] Sending connection context
0507 02:11:08.886566 (+ 59us) client_negotiation.cc:240] Negotiation successful
0507 02:11:08.886600 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":95,"threads_started":1}
I20240507 02:11:08.887022 20028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:08.884703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:44571 (local address 127.31.131.67:43123)
0507 02:11:08.884763 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:08.884766 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:08.884851 (+ 85us) server_negotiation.cc:407] Connection header received
0507 02:11:08.884918 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:08.884919 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:08.884932 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:08.884990 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:11:08.885239 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:08.885738 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:08.886453 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:08.886903 (+ 450us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:08.886913 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:08.886934 (+ 21us) server_negotiation.cc:299] Negotiation successful
0507 02:11:08.886964 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20240507 02:11:08.887055 20031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:08.884834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.131.65:53725 (local address 127.31.131.66:45981)
0507 02:11:08.884874 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:08.884876 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:08.884965 (+ 89us) server_negotiation.cc:407] Connection header received
0507 02:11:08.885014 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:08.885015 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:08.885034 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:08.885082 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0507 02:11:08.885351 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:08.885812 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:08.886507 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:08.886959 (+ 452us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:08.886965 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:08.886984 (+ 19us) server_negotiation.cc:299] Negotiation successful
0507 02:11:08.887005 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20240507 02:11:08.887305 19972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3034413da5cc4bb4bebb66e17b7edbcc" candidate_uuid: "85d58e2824624d8ba53ce817aafee478" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df60f1707b24a9092cf97422009f21e" is_pre_election: true
I20240507 02:11:08.887311 19841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3034413da5cc4bb4bebb66e17b7edbcc" candidate_uuid: "85d58e2824624d8ba53ce817aafee478" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bd96d22929e489f8d8ea8dfeadbe044" is_pre_election: true
I20240507 02:11:08.887424 19841 raft_consensus.cc:2454] T 3034413da5cc4bb4bebb66e17b7edbcc P 9bd96d22929e489f8d8ea8dfeadbe044 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85d58e2824624d8ba53ce817aafee478 in term 0.
I20240507 02:11:08.887431 19972 raft_consensus.cc:2454] T 3034413da5cc4bb4bebb66e17b7edbcc P 7df60f1707b24a9092cf97422009f21e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85d58e2824624d8ba53ce817aafee478 in term 0.
I20240507 02:11:08.887645 19644 leader_election.cc:302] T 3034413da5cc4bb4bebb66e17b7edbcc P 85d58e2824624d8ba53ce817aafee478 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7df60f1707b24a9092cf97422009f21e, 85d58e2824624d8ba53ce817aafee478; no voters:
I20240507 02:11:08.887760 20039 raft_consensus.cc:2781] T 3034413da5cc4bb4bebb66e17b7edbcc P 85d58e2824624d8ba53ce817aafee478 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 02:11:08.887827 20039 raft_consensus.cc:482] T 3034413da5cc4bb4bebb66e17b7edbcc P 85d58e2824624d8ba53ce817aafee478 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 02:11:08.887852 20039 raft_consensus.cc:3037] T 3034413da5cc4bb4bebb66e17b7edbcc P 85d58e2824624d8ba53ce817aafee478 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:11:08.888468 20039 raft_consensus.cc:504] T 3034413da5cc4bb4bebb66e17b7edbcc P 85d58e2824624d8ba53ce817aafee478 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df60f1707b24a9092cf97422009f21e" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 43123 } } peers { permanent_uuid: "9bd96d22929e489f8d8ea8dfeadbe044" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 45981 } } peers { permanent_uuid: "85d58e2824624d8ba53ce817aafee478" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 34585 } }
I20240507 02:11:08.888598 20039 leader_election.cc:288] T 3034413da5cc4bb4bebb66e17b7edbcc P 85d58e2824624d8ba53ce817aafee478 [CANDIDATE]: Term 1 election: Requested vote from peers 7df60f1707b24a9092cf97422009f21e (127.31.131.67:43123), 9bd96d22929e489f8d8ea8dfeadbe044 (127.31.131.66:45981)
I20240507 02:11:08.888739 19972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3034413da5cc4bb4bebb66e17b7edbcc" candidate_uuid: "85d58e2824624d8ba53ce817aafee478" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7df60f1707b24a9092cf97422009f21e"
I20240507 02:11:08.888783 19841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3034413da5cc4bb4bebb66e17b7edbcc" candidate_uuid: "85d58e2824624d8ba53ce817aafee478" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bd96d22929e489f8d8ea8dfeadbe044"
I20240507 02:11:08.888830 19972 raft_consensus.cc:3037] T 3034413da5cc4bb4bebb66e17b7edbcc P 7df60f1707b24a9092cf97422009f21e [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:11:08.888851 19841 raft_consensus.cc:3037] T 3034413da5cc4bb4bebb66e17b7edbcc P 9bd96d22929e489f8d8ea8dfeadbe044 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:11:08.889497 19972 raft_consensus.cc:2454] T 3034413da5cc4bb4bebb66e17b7edbcc P 7df60f1707b24a9092cf97422009f21e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85d58e2824624d8ba53ce817aafee478 in term 1.
I20240507 02:11:08.889503 19841 raft_consensus.cc:2454] T 3034413da5cc4bb4bebb66e17b7edbcc P 9bd96d22929e489f8d8ea8dfeadbe044 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85d58e2824624d8ba53ce817aafee478 in term 1.
I20240507 02:11:08.889652 19644 leader_election.cc:302] T 3034413da5cc4bb4bebb66e17b7edbcc P 85d58e2824624d8ba53ce817aafee478 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85d58e2824624d8ba53ce817aafee478, 9bd96d22929e489f8d8ea8dfeadbe044; no voters:
I20240507 02:11:08.889740 20039 raft_consensus.cc:2781] T 3034413da5cc4bb4bebb66e17b7edbcc P 85d58e2824624d8ba53ce817aafee478 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:11:08.889919 20039 raft_consensus.cc:686] T 3034413da5cc4bb4bebb66e17b7edbcc P 85d58e2824624d8ba53ce817aafee478 [term 1 LEADER]: Becoming Leader. State: Replica: 85d58e2824624d8ba53ce817aafee478, State: Running, Role: LEADER
I20240507 02:11:08.890017 20039 consensus_queue.cc:237] T 3034413da5cc4bb4bebb66e17b7edbcc P 85d58e2824624d8ba53ce817aafee478 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df60f1707b24a9092cf97422009f21e" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 43123 } } peers { permanent_uuid: "9bd96d22929e489f8d8ea8dfeadbe044" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 45981 } } peers { permanent_uuid: "85d58e2824624d8ba53ce817aafee478" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 34585 } }
I20240507 02:11:08.890769 19568 catalog_manager.cc:5484] T 3034413da5cc4bb4bebb66e17b7edbcc P 85d58e2824624d8ba53ce817aafee478 reported cstate change: term changed from 0 to 1, leader changed from <none> to 85d58e2824624d8ba53ce817aafee478 (127.31.131.65). New cstate: current_term: 1 leader_uuid: "85d58e2824624d8ba53ce817aafee478" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df60f1707b24a9092cf97422009f21e" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 43123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9bd96d22929e489f8d8ea8dfeadbe044" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 45981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85d58e2824624d8ba53ce817aafee478" member_type: VOTER last_known_addr { host: "127.31.131.65" port: 34585 } health_report { overall_health: HEALTHY } } }
I20240507 02:11:08.899515 20029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:08.897552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44748 (local address 127.31.131.65:34585)
0507 02:11:08.897595 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:08.897598 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:08.897607 (+ 9us) server_negotiation.cc:407] Connection header received
0507 02:11:08.897673 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:08.897673 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:08.897690 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:08.897757 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0507 02:11:08.897984 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:08.898403 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:08.899082 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:08.899207 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:08.899239 (+ 32us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0507 02:11:08.899316 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0507 02:11:08.899386 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:08.899440 (+ 54us) server_negotiation.cc:299] Negotiation successful
0507 02:11:08.899460 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20240507 02:11:08.900835 19841 raft_consensus.cc:1261] T 3034413da5cc4bb4bebb66e17b7edbcc P 9bd96d22929e489f8d8ea8dfeadbe044 [term 1 FOLLOWER]: Refusing update from remote peer 85d58e2824624d8ba53ce817aafee478: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 02:11:08.900843 19972 raft_consensus.cc:1261] T 3034413da5cc4bb4bebb66e17b7edbcc P 7df60f1707b24a9092cf97422009f21e [term 1 FOLLOWER]: Refusing update from remote peer 85d58e2824624d8ba53ce817aafee478: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 02:11:08.901113 20039 consensus_queue.cc:1035] T 3034413da5cc4bb4bebb66e17b7edbcc P 85d58e2824624d8ba53ce817aafee478 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bd96d22929e489f8d8ea8dfeadbe044" member_type: VOTER last_known_addr { host: "127.31.131.66" port: 45981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 02:11:08.901222 20042 consensus_queue.cc:1035] T 3034413da5cc4bb4bebb66e17b7edbcc P 85d58e2824624d8ba53ce817aafee478 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7df60f1707b24a9092cf97422009f21e" member_type: VOTER last_known_addr { host: "127.31.131.67" port: 43123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 02:11:08.903801 20046 mvcc.cc:205] Tried to move back new op lower bound from 7024836071015960576 to 7024836070973992960. Current Snapshot: MvccSnapshot[applied={T|T < 7024836071015960576}]
I20240507 02:11:08.905364 19567 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53848:
table { table_name: "test-table" } modify_external_catalogs: true
I20240507 02:11:08.905447 19567 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53848:
table { table_name: "test-table" } modify_external_catalogs: true
I20240507 02:11:08.906140 19567 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 2ce07dd3e33849769ae9d17865f24695: Sending DeleteTablet for 3 replicas of tablet 3034413da5cc4bb4bebb66e17b7edbcc
I20240507 02:11:08.906443 19952 tablet_service.cc:1514] Processing DeleteTablet for tablet 3034413da5cc4bb4bebb66e17b7edbcc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:11:08 UTC) from {username='slave'} at 127.0.0.1:33838
I20240507 02:11:08.906443 19690 tablet_service.cc:1514] Processing DeleteTablet for tablet 3034413da5cc4bb4bebb66e17b7edbcc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:11:08 UTC) from {username='slave'} at 127.0.0.1:44740
I20240507 02:11:08.906751 19821 tablet_service.cc:1514] Processing DeleteTablet for tablet 3034413da5cc4bb4bebb66e17b7edbcc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 02:11:08 UTC) from {username='slave'} at 127.0.0.1:38826
I20240507 02:11:08.906937 20056 tablet_replica.cc:330] T 3034413da5cc4bb4bebb66e17b7edbcc P 9bd96d22929e489f8d8ea8dfeadbe044: stopping tablet replica
I20240507 02:11:08.906937 20055 tablet_replica.cc:330] T 3034413da5cc4bb4bebb66e17b7edbcc P 7df60f1707b24a9092cf97422009f21e: stopping tablet replica
I20240507 02:11:08.907011 20056 raft_consensus.cc:2229] T 3034413da5cc4bb4bebb66e17b7edbcc P 9bd96d22929e489f8d8ea8dfeadbe044 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 02:11:08.907016 20055 raft_consensus.cc:2229] T 3034413da5cc4bb4bebb66e17b7edbcc P 7df60f1707b24a9092cf97422009f21e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 02:11:08.907091 20056 raft_consensus.cc:2258] T 3034413da5cc4bb4bebb66e17b7edbcc P 9bd96d22929e489f8d8ea8dfeadbe044 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 02:11:08.907091 20055 raft_consensus.cc:2258] T 3034413da5cc4bb4bebb66e17b7edbcc P 7df60f1707b24a9092cf97422009f21e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 02:11:08.907212 20054 tablet_replica.cc:330] T 3034413da5cc4bb4bebb66e17b7edbcc P 85d58e2824624d8ba53ce817aafee478: stopping tablet replica
I20240507 02:11:08.907205 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 19628
I20240507 02:11:08.907274 20054 raft_consensus.cc:2229] T 3034413da5cc4bb4bebb66e17b7edbcc P 85d58e2824624d8ba53ce817aafee478 [term 1 LEADER]: Raft consensus shutting down.
I20240507 02:11:08.907338 20054 raft_consensus.cc:2258] T 3034413da5cc4bb4bebb66e17b7edbcc P 85d58e2824624d8ba53ce817aafee478 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 02:11:08.907969 20056 ts_tablet_manager.cc:1875] T 3034413da5cc4bb4bebb66e17b7edbcc P 9bd96d22929e489f8d8ea8dfeadbe044: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 02:11:08.908134 20055 ts_tablet_manager.cc:1875] T 3034413da5cc4bb4bebb66e17b7edbcc P 7df60f1707b24a9092cf97422009f21e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 02:11:08.909335 20056 ts_tablet_manager.cc:1888] T 3034413da5cc4bb4bebb66e17b7edbcc P 9bd96d22929e489f8d8ea8dfeadbe044: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240507 02:11:08.909389 20056 log.cc:1196] T 3034413da5cc4bb4bebb66e17b7edbcc P 9bd96d22929e489f8d8ea8dfeadbe044: Deleting WAL directory at /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715047604443657-32269-0/minicluster-data/ts-1/wal/wals/3034413da5cc4bb4bebb66e17b7edbcc
I20240507 02:11:08.909384 20055 ts_tablet_manager.cc:1888] T 3034413da5cc4bb4bebb66e17b7edbcc P 7df60f1707b24a9092cf97422009f21e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240507 02:11:08.909430 20055 log.cc:1196] T 3034413da5cc4bb4bebb66e17b7edbcc P 7df60f1707b24a9092cf97422009f21e: Deleting WAL directory at /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1715047604443657-32269-0/minicluster-data/ts-2/wal/wals/3034413da5cc4bb4bebb66e17b7edbcc
I20240507 02:11:08.909638 20056 ts_tablet_manager.cc:1909] T 3034413da5cc4bb4bebb66e17b7edbcc P 9bd96d22929e489f8d8ea8dfeadbe044: Deleting consensus metadata
I20240507 02:11:08.909659 20055 ts_tablet_manager.cc:1909] T 3034413da5cc4bb4bebb66e17b7edbcc P 7df60f1707b24a9092cf97422009f21e: Deleting consensus metadata
I20240507 02:11:08.909938 19553 catalog_manager.cc:4830] TS 9bd96d22929e489f8d8ea8dfeadbe044 (127.31.131.66:45981): tablet 3034413da5cc4bb4bebb66e17b7edbcc (table test-table [id=fce06915c1804713a689e217f8378380]) successfully deleted
I20240507 02:11:08.910084 19553 catalog_manager.cc:4830] TS 7df60f1707b24a9092cf97422009f21e (127.31.131.67:43123): tablet 3034413da5cc4bb4bebb66e17b7edbcc (table test-table [id=fce06915c1804713a689e217f8378380]) successfully deleted
W20240507 02:11:08.912631 19552 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.131.65:34585 (error 108)
I20240507 02:11:08.913031 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 19759
I20240507 02:11:08.913053 20030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:08.912930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.65:34585 (local address 127.0.0.1:44754)
0507 02:11:08.912981 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:08.913016 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.65:34585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20240507 02:11:08.913534 19552 catalog_manager.cc:4573] TS 85d58e2824624d8ba53ce817aafee478 (127.31.131.65:34585): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3034413da5cc4bb4bebb66e17b7edbcc: Network error: Client connection negotiation failed: client connection to 127.31.131.65:34585: connect: Connection refused (error 111)
I20240507 02:11:08.917400 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 19890
I20240507 02:11:08.920991 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 19537
2024-05-07T02:11:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (777 ms)
[----------] 6 tests from EncryptionPolicyTest (7371 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1715047604443657-32269-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 07 02:11:08 dist-test-slave-v0bg krb5kdc[20060](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 07 02:11:08 dist-test-slave-v0bg krb5kdc[20060](info): set up 2 sockets
May 07 02:11:08 dist-test-slave-v0bg krb5kdc[20060](info): commencing operation
krb5kdc: starting...
W20240507 02:11:09.799055 32269 mini_kdc.cc:121] Time spent starting KDC: real 0.861s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1715047604443657-32269-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1715047604443657-32269-0/krb5kdc/test-user.keytab.
May 07 02:11:09 dist-test-slave-v0bg krb5kdc[20060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-07T02:11:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-07T02:11:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1715047604443657-32269-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1715047604443657-32269-0/krb5kdc/kudu_127.0.0.1.keytab.
I20240507 02:11:09.904848 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1715047604443657-32269-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1715047604443657-32269-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1715047604443657-32269-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33423
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1715047604443657-32269-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33423
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:11:09.982306 20073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:11:09.982477 20073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:11:09.982501 20073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:11:09.983764 20073 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240507 02:11:09.983811 20073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:11:09.983826 20073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:11:09.983853 20073 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240507 02:11:09.983879 20073 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240507 02:11:09.985426 20073 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1715047604443657-32269-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1715047604443657-32269-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33423
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1715047604443657-32269-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1715047604443657-32269-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:11:09.985621 20073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:11:09.985788 20073 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:11:09.989338 20079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:11:09.989367 20078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:09.989507 20073 server_base.cc:1034] running on GCE node
W20240507 02:11:09.989518 20081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:09.989746 20073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:11:09.989933 20073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:11:09.991051 20073 hybrid_clock.cc:648] HybridClock initialized: now 1715047869991044 us; error 25 us; skew 500 ppm
May 07 02:11:09 dist-test-slave-v0bg krb5kdc[20060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047869, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:11:09.993184 20073 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20240507 02:11:09.993573 20073 webserver.cc:460] Webserver started at http://127.0.0.1:43889/ using document root <none> and password file <none>
I20240507 02:11:09.993727 20073 fs_manager.cc:359] Metadata directory not provided
I20240507 02:11:09.993768 20073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:11:09.993871 20073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:11:09.994825 20073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1715047604443657-32269-0/minicluster-data/master-0/data/instance:
uuid: "2e567cafec15462e973504a51a44898f"
format_stamp: "Formatted at 2024-05-07 02:11:09 on dist-test-slave-v0bg"
server_key: "3e0d1119c7a5c17521eff7cbb77f94c5"
server_key_iv: "37395f84b81e11638abc1c1da6fc2dcd"
server_key_version: "encryptionkey@0"
I20240507 02:11:09.995106 20073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1715047604443657-32269-0/minicluster-data/master-0/wal/instance:
uuid: "2e567cafec15462e973504a51a44898f"
format_stamp: "Formatted at 2024-05-07 02:11:09 on dist-test-slave-v0bg"
server_key: "3e0d1119c7a5c17521eff7cbb77f94c5"
server_key_iv: "37395f84b81e11638abc1c1da6fc2dcd"
server_key_version: "encryptionkey@0"
I20240507 02:11:09.996184 20073 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240507 02:11:09.996766 20088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:09.996915 20073 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240507 02:11:09.996985 20073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1715047604443657-32269-0/minicluster-data/master-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1715047604443657-32269-0/minicluster-data/master-0/wal
uuid: "2e567cafec15462e973504a51a44898f"
format_stamp: "Formatted at 2024-05-07 02:11:09 on dist-test-slave-v0bg"
server_key: "3e0d1119c7a5c17521eff7cbb77f94c5"
server_key_iv: "37395f84b81e11638abc1c1da6fc2dcd"
server_key_version: "encryptionkey@0"
I20240507 02:11:09.997045 20073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1715047604443657-32269-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1715047604443657-32269-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1715047604443657-32269-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:11:10.031275 20073 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:11:10.032714 20073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:11:10.032825 20073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:11:10.036259 20073 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33423
I20240507 02:11:10.036284 20140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33423 every 8 connection(s)
I20240507 02:11:10.036566 20073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
I20240507 02:11:10.037186 20141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:11:10.038697 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 20073
I20240507 02:11:10.038803 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1715047604443657-32269-0/minicluster-data/master-0/wal/instance
I20240507 02:11:10.038998 32269 external_mini_cluster.cc:1428] Setting key 14273b33ed8feb5f0bc5dde19d55beef
I20240507 02:11:10.039160 20141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e567cafec15462e973504a51a44898f: Bootstrap starting.
I20240507 02:11:10.039708 20141 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e567cafec15462e973504a51a44898f: Neither blocks nor log segments found. Creating new log.
I20240507 02:11:10.039945 20141 log.cc:824] T 00000000000000000000000000000000 P 2e567cafec15462e973504a51a44898f: Log is configured to *not* fsync() on all Append() calls
I20240507 02:11:10.040478 20141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e567cafec15462e973504a51a44898f: No bootstrap required, opened a new log
I20240507 02:11:10.041865 20141 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e567cafec15462e973504a51a44898f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e567cafec15462e973504a51a44898f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33423 } }
I20240507 02:11:10.041967 20141 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e567cafec15462e973504a51a44898f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:11:10.041989 20141 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e567cafec15462e973504a51a44898f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e567cafec15462e973504a51a44898f, State: Initialized, Role: FOLLOWER
I20240507 02:11:10.042071 20141 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e567cafec15462e973504a51a44898f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e567cafec15462e973504a51a44898f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33423 } }
I20240507 02:11:10.042130 20141 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e567cafec15462e973504a51a44898f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 02:11:10.042158 20141 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e567cafec15462e973504a51a44898f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 02:11:10.042199 20141 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e567cafec15462e973504a51a44898f [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:11:10.042699 20141 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e567cafec15462e973504a51a44898f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e567cafec15462e973504a51a44898f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33423 } }
I20240507 02:11:10.042801 20141 leader_election.cc:302] T 00000000000000000000000000000000 P 2e567cafec15462e973504a51a44898f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e567cafec15462e973504a51a44898f; no voters:
I20240507 02:11:10.042941 20141 leader_election.cc:288] T 00000000000000000000000000000000 P 2e567cafec15462e973504a51a44898f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240507 02:11:10.042973 20146 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e567cafec15462e973504a51a44898f [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:11:10.043081 20146 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e567cafec15462e973504a51a44898f [term 1 LEADER]: Becoming Leader. State: Replica: 2e567cafec15462e973504a51a44898f, State: Running, Role: LEADER
I20240507 02:11:10.043159 20141 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e567cafec15462e973504a51a44898f [sys.catalog]: configured and running, proceeding with master startup.
I20240507 02:11:10.043210 20146 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e567cafec15462e973504a51a44898f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e567cafec15462e973504a51a44898f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33423 } }
I20240507 02:11:10.043524 20147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e567cafec15462e973504a51a44898f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e567cafec15462e973504a51a44898f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e567cafec15462e973504a51a44898f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33423 } } }
I20240507 02:11:10.043591 20147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e567cafec15462e973504a51a44898f [sys.catalog]: This master's current role is: LEADER
I20240507 02:11:10.043753 20148 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e567cafec15462e973504a51a44898f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e567cafec15462e973504a51a44898f. Latest consensus state: current_term: 1 leader_uuid: "2e567cafec15462e973504a51a44898f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e567cafec15462e973504a51a44898f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33423 } } }
I20240507 02:11:10.044139 20148 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e567cafec15462e973504a51a44898f [sys.catalog]: This master's current role is: LEADER
I20240507 02:11:10.044289 20150 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
May 07 02:11:10 dist-test-slave-v0bg krb5kdc[20060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20240507 02:11:10.044721 20150 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240507 02:11:10.046082 20150 catalog_manager.cc:1347] Generated new cluster ID: 7512cc29a832441b975758cf455828d2
I20240507 02:11:10.046451 20150 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240507 02:11:10.048725 20144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:10.039462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57005 (local address 127.0.0.1:33423)
0507 02:11:10.039629 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:10.039638 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:10.039653 (+ 15us) server_negotiation.cc:407] Connection header received
0507 02:11:10.040099 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:10.040105 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:10.040146 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:10.040265 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:10.041012 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:10.041778 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:10.042433 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:10.042673 (+ 240us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:10.045205 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:10.045211 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:10.045212 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:11:10.045229 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:10.047213 (+ 1984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:10.047578 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:10.047581 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:10.047581 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:10.047630 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:10.047792 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:10.047792 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:10.047792 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:10.047981 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:10.048110 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:10.048477 (+ 367us) server_negotiation.cc:299] Negotiation successful
0507 02:11:10.048563 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":83,"threads_started":1}
I20240507 02:11:10.049489 32269 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240507 02:11:10.056918 20144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:10.050675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40057 (local address 127.0.0.1:33423)
0507 02:11:10.050709 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:10.050710 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:10.050892 (+ 182us) server_negotiation.cc:407] Connection header received
0507 02:11:10.050953 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:10.050954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:10.050962 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:10.051003 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:10.051769 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:10.052423 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:10.053087 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:10.053234 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:10.054411 (+ 1177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:10.054414 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:10.054414 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:11:10.054419 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:10.055917 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:10.056293 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:10.056294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:10.056294 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:10.056311 (+ 17us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:10.056480 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:10.056480 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:10.056480 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:10.056559 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:10.056634 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:10.056791 (+ 157us) server_negotiation.cc:299] Negotiation successful
0507 02:11:10.056842 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20240507 02:11:10.062669 20150 catalog_manager.cc:1370] Generated new certificate authority record
I20240507 02:11:10.063117 20150 catalog_manager.cc:1504] Loading token signing keys...
I20240507 02:11:10.071157 20150 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 2e567cafec15462e973504a51a44898f: Generated new TSK 0
I20240507 02:11:10.071286 20150 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240507 02:11:10.075150 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 20073
W20240507 02:11:10.078907 20058 connection.cc:535] client connection to 127.0.0.1:33423 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33423): Connection reset by peer (error 104)
2024-05-07T02:11:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1153 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1715047604443657-32269-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 07 02:11:10 dist-test-slave-v0bg krb5kdc[20172](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 07 02:11:10 dist-test-slave-v0bg krb5kdc[20172](info): set up 2 sockets
May 07 02:11:10 dist-test-slave-v0bg krb5kdc[20172](info): commencing operation
krb5kdc: starting...
W20240507 02:11:10.963112 32269 mini_kdc.cc:121] Time spent starting KDC: real 0.871s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1715047604443657-32269-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1715047604443657-32269-0/krb5kdc/test-user.keytab.
May 07 02:11:11 dist-test-slave-v0bg krb5kdc[20172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-07T02:11:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-07T02:11:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1715047604443657-32269-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1715047604443657-32269-0/krb5kdc/kudu_127.0.0.1.keytab.
I20240507 02:11:11.059667 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1715047604443657-32269-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1715047604443657-32269-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1715047604443657-32269-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38081
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1715047604443657-32269-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38081
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:11:11.132602 20185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:11:11.132764 20185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:11:11.132784 20185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:11:11.133999 20185 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240507 02:11:11.134037 20185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:11:11.134050 20185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:11:11.134073 20185 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240507 02:11:11.134095 20185 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240507 02:11:11.135648 20185 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1715047604443657-32269-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1715047604443657-32269-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38081
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1715047604443657-32269-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38081
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1715047604443657-32269-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:11:11.135818 20185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:11:11.135974 20185 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240507 02:11:11.139381 20185 server_base.cc:1034] running on GCE node
W20240507 02:11:11.139353 20191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:11:11.139482 20190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:11:11.139338 20193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:11.139727 20185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:11:11.139904 20185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:11:11.141026 20185 hybrid_clock.cc:648] HybridClock initialized: now 1715047871141015 us; error 29 us; skew 500 ppm
May 07 02:11:11 dist-test-slave-v0bg krb5kdc[20172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047871, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:11:11.148461 20185 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20240507 02:11:11.148787 20185 webserver.cc:460] Webserver started at http://127.0.0.1:43895/ using document root <none> and password file <none>
I20240507 02:11:11.148911 20185 fs_manager.cc:359] Metadata directory not provided
I20240507 02:11:11.148941 20185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:11:11.149013 20185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:11:11.149817 20185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1715047604443657-32269-0/minicluster-data/master-0/data/instance:
uuid: "4b99b8e8c07f4dd1b70effa6298bff39"
format_stamp: "Formatted at 2024-05-07 02:11:11 on dist-test-slave-v0bg"
server_key: "b2f3298cda87f3c68c86470fc890b739"
server_key_iv: "b2421dbed4872dbedc590d67eef613ae"
server_key_version: "encryptionkey@0"
I20240507 02:11:11.150043 20185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1715047604443657-32269-0/minicluster-data/master-0/wal/instance:
uuid: "4b99b8e8c07f4dd1b70effa6298bff39"
format_stamp: "Formatted at 2024-05-07 02:11:11 on dist-test-slave-v0bg"
server_key: "b2f3298cda87f3c68c86470fc890b739"
server_key_iv: "b2421dbed4872dbedc590d67eef613ae"
server_key_version: "encryptionkey@0"
I20240507 02:11:11.151037 20185 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240507 02:11:11.151599 20200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:11.151762 20185 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240507 02:11:11.151825 20185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1715047604443657-32269-0/minicluster-data/master-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1715047604443657-32269-0/minicluster-data/master-0/wal
uuid: "4b99b8e8c07f4dd1b70effa6298bff39"
format_stamp: "Formatted at 2024-05-07 02:11:11 on dist-test-slave-v0bg"
server_key: "b2f3298cda87f3c68c86470fc890b739"
server_key_iv: "b2421dbed4872dbedc590d67eef613ae"
server_key_version: "encryptionkey@0"
I20240507 02:11:11.151871 20185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1715047604443657-32269-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1715047604443657-32269-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1715047604443657-32269-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:11:11.167204 20185 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:11:11.168540 20185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:11:11.168622 20185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:11:11.171627 20185 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38081
I20240507 02:11:11.171666 20253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38081 every 8 connection(s)
I20240507 02:11:11.171895 20185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
I20240507 02:11:11.172482 20254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:11:11.172776 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 20185
I20240507 02:11:11.172871 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1715047604443657-32269-0/minicluster-data/master-0/wal/instance
I20240507 02:11:11.173022 32269 external_mini_cluster.cc:1428] Setting key 98d903a6f0add9eca6ac6d25e2ba9d13
I20240507 02:11:11.174790 20254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b99b8e8c07f4dd1b70effa6298bff39: Bootstrap starting.
I20240507 02:11:11.175242 20254 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b99b8e8c07f4dd1b70effa6298bff39: Neither blocks nor log segments found. Creating new log.
I20240507 02:11:11.175467 20254 log.cc:824] T 00000000000000000000000000000000 P 4b99b8e8c07f4dd1b70effa6298bff39: Log is configured to *not* fsync() on all Append() calls
I20240507 02:11:11.176111 20254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b99b8e8c07f4dd1b70effa6298bff39: No bootstrap required, opened a new log
I20240507 02:11:11.177042 20257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:11.173548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38287 (local address 127.0.0.1:38081)
0507 02:11:11.173693 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:11.173701 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:11.173715 (+ 14us) server_negotiation.cc:407] Connection header received
0507 02:11:11.174045 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:11.174049 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:11.174084 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:11.174185 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:11.174651 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:11.175354 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:11.176030 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:11.176195 (+ 165us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:11.176552 (+ 357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:11.176558 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:11.176558 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0507 02:11:11.176695 (+ 137us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:11.176710 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0507 02:11:11.176716 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:11.176778 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:11.176881 (+ 103us) server_negotiation.cc:299] Negotiation successful
0507 02:11:11.176900 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":77,"threads_started":1}
I20240507 02:11:11.177942 32269 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240507 02:11:11.178164 20254 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b99b8e8c07f4dd1b70effa6298bff39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b99b8e8c07f4dd1b70effa6298bff39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38081 } }
I20240507 02:11:11.178267 20254 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b99b8e8c07f4dd1b70effa6298bff39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:11:11.178282 20254 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b99b8e8c07f4dd1b70effa6298bff39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b99b8e8c07f4dd1b70effa6298bff39, State: Initialized, Role: FOLLOWER
I20240507 02:11:11.178356 20254 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b99b8e8c07f4dd1b70effa6298bff39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b99b8e8c07f4dd1b70effa6298bff39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38081 } }
I20240507 02:11:11.178406 20254 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b99b8e8c07f4dd1b70effa6298bff39 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 02:11:11.178423 20254 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b99b8e8c07f4dd1b70effa6298bff39 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 02:11:11.178463 20254 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b99b8e8c07f4dd1b70effa6298bff39 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:11:11.179004 20254 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b99b8e8c07f4dd1b70effa6298bff39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b99b8e8c07f4dd1b70effa6298bff39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38081 } }
I20240507 02:11:11.179172 20254 leader_election.cc:302] T 00000000000000000000000000000000 P 4b99b8e8c07f4dd1b70effa6298bff39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b99b8e8c07f4dd1b70effa6298bff39; no voters:
I20240507 02:11:11.179307 20254 leader_election.cc:288] T 00000000000000000000000000000000 P 4b99b8e8c07f4dd1b70effa6298bff39 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240507 02:11:11.179337 20264 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b99b8e8c07f4dd1b70effa6298bff39 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:11:11.179482 20264 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b99b8e8c07f4dd1b70effa6298bff39 [term 1 LEADER]: Becoming Leader. State: Replica: 4b99b8e8c07f4dd1b70effa6298bff39, State: Running, Role: LEADER
I20240507 02:11:11.179514 20254 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b99b8e8c07f4dd1b70effa6298bff39 [sys.catalog]: configured and running, proceeding with master startup.
I20240507 02:11:11.179576 20264 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b99b8e8c07f4dd1b70effa6298bff39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b99b8e8c07f4dd1b70effa6298bff39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38081 } }
I20240507 02:11:11.179832 20264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b99b8e8c07f4dd1b70effa6298bff39 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b99b8e8c07f4dd1b70effa6298bff39. Latest consensus state: current_term: 1 leader_uuid: "4b99b8e8c07f4dd1b70effa6298bff39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b99b8e8c07f4dd1b70effa6298bff39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38081 } } }
I20240507 02:11:11.180025 20264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b99b8e8c07f4dd1b70effa6298bff39 [sys.catalog]: This master's current role is: LEADER
I20240507 02:11:11.179955 20265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b99b8e8c07f4dd1b70effa6298bff39 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b99b8e8c07f4dd1b70effa6298bff39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b99b8e8c07f4dd1b70effa6298bff39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38081 } } }
I20240507 02:11:11.180208 20265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b99b8e8c07f4dd1b70effa6298bff39 [sys.catalog]: This master's current role is: LEADER
I20240507 02:11:11.180347 20273 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 02:11:11.180807 20273 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240507 02:11:11.181797 20257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:11.179116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47103 (local address 127.0.0.1:38081)
0507 02:11:11.179164 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:11.179165 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:11.179255 (+ 90us) server_negotiation.cc:407] Connection header received
0507 02:11:11.179308 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:11.179309 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:11.179317 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:11.179361 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:11.179787 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:11.180345 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:11.181149 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:11.181301 (+ 152us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:11.181583 (+ 282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:11.181586 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:11.181586 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0507 02:11:11.181590 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:11.181601 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0507 02:11:11.181605 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:11.181652 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:11.181747 (+ 95us) server_negotiation.cc:299] Negotiation successful
0507 02:11:11.181755 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20240507 02:11:11.182821 20273 catalog_manager.cc:1347] Generated new cluster ID: ec77ff5edcec4a4e98ddc2f97ef4fd44
I20240507 02:11:11.182858 20273 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240507 02:11:11.202327 20273 catalog_manager.cc:1370] Generated new certificate authority record
I20240507 02:11:11.202832 20273 catalog_manager.cc:1504] Loading token signing keys...
I20240507 02:11:11.208689 20273 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 4b99b8e8c07f4dd1b70effa6298bff39: Generated new TSK 0
I20240507 02:11:11.208828 20273 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240507 02:11:11.219558 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 20185
W20240507 02:11:11.223424 20170 connection.cc:535] client connection to 127.0.0.1:38081 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38081): Connection reset by peer (error 104)
2024-05-07T02:11:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1144 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1715047604443657-32269-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 07 02:11:11 dist-test-slave-v0bg krb5kdc[20285](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 07 02:11:11 dist-test-slave-v0bg krb5kdc[20285](info): set up 2 sockets
May 07 02:11:11 dist-test-slave-v0bg krb5kdc[20285](info): commencing operation
krb5kdc: starting...
W20240507 02:11:12.115119 32269 mini_kdc.cc:121] Time spent starting KDC: real 0.879s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1715047604443657-32269-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1715047604443657-32269-0/krb5kdc/test-user.keytab.
May 07 02:11:12 dist-test-slave-v0bg krb5kdc[20285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-07T02:11:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-07T02:11:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1715047604443657-32269-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1715047604443657-32269-0/krb5kdc/kudu_127.0.0.1.keytab.
I20240507 02:11:12.213466 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1715047604443657-32269-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1715047604443657-32269-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1715047604443657-32269-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46029
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1715047604443657-32269-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46029
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:11:12.288164 20298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:11:12.288323 20298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:11:12.288352 20298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:11:12.289559 20298 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240507 02:11:12.289598 20298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:11:12.289616 20298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:11:12.289631 20298 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240507 02:11:12.289645 20298 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240507 02:11:12.291169 20298 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1715047604443657-32269-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1715047604443657-32269-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46029
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1715047604443657-32269-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1715047604443657-32269-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:11:12.291349 20298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:11:12.291494 20298 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:11:12.294792 20306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:11:12.294824 20304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:12.294907 20298 server_base.cc:1034] running on GCE node
W20240507 02:11:12.294957 20303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:12.295210 20298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:11:12.295408 20298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:11:12.296531 20298 hybrid_clock.cc:648] HybridClock initialized: now 1715047872296514 us; error 32 us; skew 500 ppm
May 07 02:11:12 dist-test-slave-v0bg krb5kdc[20285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047872, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:11:12.298523 20298 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20240507 02:11:12.298871 20298 webserver.cc:460] Webserver started at http://127.0.0.1:37207/ using document root <none> and password file <none>
I20240507 02:11:12.299016 20298 fs_manager.cc:359] Metadata directory not provided
I20240507 02:11:12.299054 20298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:11:12.299144 20298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:11:12.299952 20298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1715047604443657-32269-0/minicluster-data/master-0/data/instance:
uuid: "7930e40f6d3a48f890f6f7c4565a0a25"
format_stamp: "Formatted at 2024-05-07 02:11:12 on dist-test-slave-v0bg"
server_key: "63ab6906534963f7ee27f96c714351fa"
server_key_iv: "5b6d7f663f951cda603321fdbd710616"
server_key_version: "encryptionkey@0"
I20240507 02:11:12.300210 20298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1715047604443657-32269-0/minicluster-data/master-0/wal/instance:
uuid: "7930e40f6d3a48f890f6f7c4565a0a25"
format_stamp: "Formatted at 2024-05-07 02:11:12 on dist-test-slave-v0bg"
server_key: "63ab6906534963f7ee27f96c714351fa"
server_key_iv: "5b6d7f663f951cda603321fdbd710616"
server_key_version: "encryptionkey@0"
I20240507 02:11:12.301193 20298 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240507 02:11:12.301769 20313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:12.301923 20298 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240507 02:11:12.301991 20298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1715047604443657-32269-0/minicluster-data/master-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1715047604443657-32269-0/minicluster-data/master-0/wal
uuid: "7930e40f6d3a48f890f6f7c4565a0a25"
format_stamp: "Formatted at 2024-05-07 02:11:12 on dist-test-slave-v0bg"
server_key: "63ab6906534963f7ee27f96c714351fa"
server_key_iv: "5b6d7f663f951cda603321fdbd710616"
server_key_version: "encryptionkey@0"
I20240507 02:11:12.302044 20298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1715047604443657-32269-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1715047604443657-32269-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1715047604443657-32269-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:11:12.305127 20298 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:11:12.306596 20298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:11:12.306690 20298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:11:12.309691 20298 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46029
I20240507 02:11:12.309717 20365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46029 every 8 connection(s)
I20240507 02:11:12.309944 20298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
I20240507 02:11:12.310673 20366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:11:12.312381 20366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7930e40f6d3a48f890f6f7c4565a0a25: Bootstrap starting.
I20240507 02:11:12.312856 20366 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7930e40f6d3a48f890f6f7c4565a0a25: Neither blocks nor log segments found. Creating new log.
I20240507 02:11:12.313074 20366 log.cc:824] T 00000000000000000000000000000000 P 7930e40f6d3a48f890f6f7c4565a0a25: Log is configured to *not* fsync() on all Append() calls
I20240507 02:11:12.313602 20366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7930e40f6d3a48f890f6f7c4565a0a25: No bootstrap required, opened a new log
I20240507 02:11:12.314940 20366 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7930e40f6d3a48f890f6f7c4565a0a25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7930e40f6d3a48f890f6f7c4565a0a25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46029 } }
I20240507 02:11:12.315043 20366 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7930e40f6d3a48f890f6f7c4565a0a25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:11:12.315058 20366 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7930e40f6d3a48f890f6f7c4565a0a25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7930e40f6d3a48f890f6f7c4565a0a25, State: Initialized, Role: FOLLOWER
I20240507 02:11:12.315148 20366 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7930e40f6d3a48f890f6f7c4565a0a25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7930e40f6d3a48f890f6f7c4565a0a25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46029 } }
I20240507 02:11:12.315204 20366 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7930e40f6d3a48f890f6f7c4565a0a25 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 02:11:12.315223 20366 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7930e40f6d3a48f890f6f7c4565a0a25 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 02:11:12.315266 20366 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7930e40f6d3a48f890f6f7c4565a0a25 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:11:12.315723 20366 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7930e40f6d3a48f890f6f7c4565a0a25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7930e40f6d3a48f890f6f7c4565a0a25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46029 } }
I20240507 02:11:12.315804 20366 leader_election.cc:302] T 00000000000000000000000000000000 P 7930e40f6d3a48f890f6f7c4565a0a25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7930e40f6d3a48f890f6f7c4565a0a25; no voters:
I20240507 02:11:12.315929 20366 leader_election.cc:288] T 00000000000000000000000000000000 P 7930e40f6d3a48f890f6f7c4565a0a25 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240507 02:11:12.315973 20369 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7930e40f6d3a48f890f6f7c4565a0a25 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:11:12.316114 20369 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7930e40f6d3a48f890f6f7c4565a0a25 [term 1 LEADER]: Becoming Leader. State: Replica: 7930e40f6d3a48f890f6f7c4565a0a25, State: Running, Role: LEADER
I20240507 02:11:12.316125 20366 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7930e40f6d3a48f890f6f7c4565a0a25 [sys.catalog]: configured and running, proceeding with master startup.
I20240507 02:11:12.316202 20369 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7930e40f6d3a48f890f6f7c4565a0a25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7930e40f6d3a48f890f6f7c4565a0a25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46029 } }
I20240507 02:11:12.316366 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 20298
I20240507 02:11:12.316498 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1715047604443657-32269-0/minicluster-data/master-0/wal/instance
I20240507 02:11:12.316651 32269 external_mini_cluster.cc:1428] Setting key 4981432c796349ddc40dd3465b697bd0
I20240507 02:11:12.316830 20370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7930e40f6d3a48f890f6f7c4565a0a25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7930e40f6d3a48f890f6f7c4565a0a25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7930e40f6d3a48f890f6f7c4565a0a25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46029 } } }
I20240507 02:11:12.316954 20370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7930e40f6d3a48f890f6f7c4565a0a25 [sys.catalog]: This master's current role is: LEADER
I20240507 02:11:12.316813 20371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7930e40f6d3a48f890f6f7c4565a0a25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7930e40f6d3a48f890f6f7c4565a0a25. Latest consensus state: current_term: 1 leader_uuid: "7930e40f6d3a48f890f6f7c4565a0a25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7930e40f6d3a48f890f6f7c4565a0a25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46029 } } }
I20240507 02:11:12.317137 20371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7930e40f6d3a48f890f6f7c4565a0a25 [sys.catalog]: This master's current role is: LEADER
I20240507 02:11:12.317279 20379 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 02:11:12.318001 20379 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240507 02:11:12.318836 20380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:12.317161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36131 (local address 127.0.0.1:46029)
0507 02:11:12.317416 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:12.317427 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:12.317447 (+ 20us) server_negotiation.cc:407] Connection header received
0507 02:11:12.317549 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:12.317555 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:12.317589 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:12.317729 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:12.318330 (+ 601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:12.318333 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:12.318333 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0507 02:11:12.318337 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:12.318357 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0507 02:11:12.318364 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:12.318439 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:12.318653 (+ 214us) server_negotiation.cc:299] Negotiation successful
0507 02:11:12.318685 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":57,"threads_started":1}
I20240507 02:11:12.319190 20379 catalog_manager.cc:1347] Generated new cluster ID: 2ffef5fa416c4d27b434d452e3d19081
I20240507 02:11:12.319231 20379 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240507 02:11:12.319656 32269 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240507 02:11:12.327714 20380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:12.326801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36779 (local address 127.0.0.1:46029)
0507 02:11:12.326830 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:12.326832 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:12.326948 (+ 116us) server_negotiation.cc:407] Connection header received
0507 02:11:12.327012 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:12.327013 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:12.327017 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:12.327059 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:12.327538 (+ 479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:12.327539 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:12.327540 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0507 02:11:12.327543 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:12.327555 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0507 02:11:12.327558 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:12.327601 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:12.327648 (+ 47us) server_negotiation.cc:299] Negotiation successful
0507 02:11:12.327659 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20240507 02:11:12.335707 20379 catalog_manager.cc:1370] Generated new certificate authority record
I20240507 02:11:12.335829 20379 catalog_manager.cc:1504] Loading token signing keys...
I20240507 02:11:12.345094 20379 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 7930e40f6d3a48f890f6f7c4565a0a25: Generated new TSK 0
I20240507 02:11:12.345235 20379 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240507 02:11:12.349687 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 20298
2024-05-07T02:11:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1129 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1715047604443657-32269-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 07 02:11:12 dist-test-slave-v0bg krb5kdc[20397](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 07 02:11:12 dist-test-slave-v0bg krb5kdc[20397](info): set up 2 sockets
May 07 02:11:12 dist-test-slave-v0bg krb5kdc[20397](info): commencing operation
krb5kdc: starting...
W20240507 02:11:13.227205 32269 mini_kdc.cc:121] Time spent starting KDC: real 0.861s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1715047604443657-32269-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1715047604443657-32269-0/krb5kdc/test-user.keytab.
May 07 02:11:13 dist-test-slave-v0bg krb5kdc[20397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-07T02:11:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-07T02:11:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1715047604443657-32269-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1715047604443657-32269-0/krb5kdc/kudu_127.0.0.1.keytab.
I20240507 02:11:13.326903 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1715047604443657-32269-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1715047604443657-32269-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1715047604443657-32269-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42233
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1715047604443657-32269-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42233
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:11:13.399590 20410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:11:13.399755 20410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:11:13.399783 20410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:11:13.401011 20410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240507 02:11:13.401058 20410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:11:13.401077 20410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:11:13.401091 20410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240507 02:11:13.401104 20410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240507 02:11:13.402669 20410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1715047604443657-32269-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1715047604443657-32269-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42233
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1715047604443657-32269-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1715047604443657-32269-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:11:13.402844 20410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:11:13.402988 20410 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:11:13.406497 20418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:13.406592 20410 server_base.cc:1034] running on GCE node
W20240507 02:11:13.406628 20415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:11:13.406741 20416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:13.406962 20410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:11:13.407171 20410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:11:13.408288 20410 hybrid_clock.cc:648] HybridClock initialized: now 1715047873408283 us; error 29 us; skew 500 ppm
May 07 02:11:13 dist-test-slave-v0bg krb5kdc[20397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047873, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:11:13.410394 20410 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20240507 02:11:13.410802 20410 webserver.cc:460] Webserver started at http://127.0.0.1:44053/ using document root <none> and password file <none>
I20240507 02:11:13.410955 20410 fs_manager.cc:359] Metadata directory not provided
I20240507 02:11:13.410995 20410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:11:13.411089 20410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:11:13.411974 20410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1715047604443657-32269-0/minicluster-data/master-0/data/instance:
uuid: "034af77601684d0388c1f223fc837629"
format_stamp: "Formatted at 2024-05-07 02:11:13 on dist-test-slave-v0bg"
server_key: "3341a63248798214adb83c749d15650b"
server_key_iv: "ac6b03d48e231654e3ea0ae05f4bc6f6"
server_key_version: "encryptionkey@0"
I20240507 02:11:13.412242 20410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1715047604443657-32269-0/minicluster-data/master-0/wal/instance:
uuid: "034af77601684d0388c1f223fc837629"
format_stamp: "Formatted at 2024-05-07 02:11:13 on dist-test-slave-v0bg"
server_key: "3341a63248798214adb83c749d15650b"
server_key_iv: "ac6b03d48e231654e3ea0ae05f4bc6f6"
server_key_version: "encryptionkey@0"
I20240507 02:11:13.413245 20410 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240507 02:11:13.413810 20425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:13.413951 20410 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240507 02:11:13.414017 20410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1715047604443657-32269-0/minicluster-data/master-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1715047604443657-32269-0/minicluster-data/master-0/wal
uuid: "034af77601684d0388c1f223fc837629"
format_stamp: "Formatted at 2024-05-07 02:11:13 on dist-test-slave-v0bg"
server_key: "3341a63248798214adb83c749d15650b"
server_key_iv: "ac6b03d48e231654e3ea0ae05f4bc6f6"
server_key_version: "encryptionkey@0"
I20240507 02:11:13.414072 20410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1715047604443657-32269-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1715047604443657-32269-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1715047604443657-32269-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:11:13.446089 20410 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:11:13.447639 20410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:11:13.447746 20410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:11:13.450951 20410 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42233
I20240507 02:11:13.450994 20477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42233 every 8 connection(s)
I20240507 02:11:13.451272 20410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
I20240507 02:11:13.451792 20478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:11:13.453780 20478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 034af77601684d0388c1f223fc837629: Bootstrap starting.
I20240507 02:11:13.454308 20478 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 034af77601684d0388c1f223fc837629: Neither blocks nor log segments found. Creating new log.
I20240507 02:11:13.454535 20478 log.cc:824] T 00000000000000000000000000000000 P 034af77601684d0388c1f223fc837629: Log is configured to *not* fsync() on all Append() calls
I20240507 02:11:13.455147 20478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 034af77601684d0388c1f223fc837629: No bootstrap required, opened a new log
I20240507 02:11:13.456483 20478 raft_consensus.cc:348] T 00000000000000000000000000000000 P 034af77601684d0388c1f223fc837629 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "034af77601684d0388c1f223fc837629" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42233 } }
I20240507 02:11:13.456604 20478 raft_consensus.cc:374] T 00000000000000000000000000000000 P 034af77601684d0388c1f223fc837629 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:11:13.456625 20478 raft_consensus.cc:729] T 00000000000000000000000000000000 P 034af77601684d0388c1f223fc837629 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 034af77601684d0388c1f223fc837629, State: Initialized, Role: FOLLOWER
I20240507 02:11:13.456728 20478 consensus_queue.cc:260] T 00000000000000000000000000000000 P 034af77601684d0388c1f223fc837629 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "034af77601684d0388c1f223fc837629" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42233 } }
I20240507 02:11:13.456782 20478 raft_consensus.cc:388] T 00000000000000000000000000000000 P 034af77601684d0388c1f223fc837629 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 02:11:13.456808 20478 raft_consensus.cc:482] T 00000000000000000000000000000000 P 034af77601684d0388c1f223fc837629 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 02:11:13.456849 20478 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 034af77601684d0388c1f223fc837629 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:11:13.457342 20478 raft_consensus.cc:504] T 00000000000000000000000000000000 P 034af77601684d0388c1f223fc837629 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "034af77601684d0388c1f223fc837629" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42233 } }
I20240507 02:11:13.457432 20478 leader_election.cc:302] T 00000000000000000000000000000000 P 034af77601684d0388c1f223fc837629 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 034af77601684d0388c1f223fc837629; no voters:
I20240507 02:11:13.457567 20478 leader_election.cc:288] T 00000000000000000000000000000000 P 034af77601684d0388c1f223fc837629 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240507 02:11:13.457607 20481 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 034af77601684d0388c1f223fc837629 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:11:13.457754 20481 raft_consensus.cc:686] T 00000000000000000000000000000000 P 034af77601684d0388c1f223fc837629 [term 1 LEADER]: Becoming Leader. State: Replica: 034af77601684d0388c1f223fc837629, State: Running, Role: LEADER
I20240507 02:11:13.457839 20481 consensus_queue.cc:237] T 00000000000000000000000000000000 P 034af77601684d0388c1f223fc837629 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "034af77601684d0388c1f223fc837629" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42233 } }
I20240507 02:11:13.458106 20478 sys_catalog.cc:564] T 00000000000000000000000000000000 P 034af77601684d0388c1f223fc837629 [sys.catalog]: configured and running, proceeding with master startup.
I20240507 02:11:13.458201 20482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 034af77601684d0388c1f223fc837629 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "034af77601684d0388c1f223fc837629" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "034af77601684d0388c1f223fc837629" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42233 } } }
I20240507 02:11:13.458284 20482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 034af77601684d0388c1f223fc837629 [sys.catalog]: This master's current role is: LEADER
I20240507 02:11:13.458426 20483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 034af77601684d0388c1f223fc837629 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 034af77601684d0388c1f223fc837629. Latest consensus state: current_term: 1 leader_uuid: "034af77601684d0388c1f223fc837629" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "034af77601684d0388c1f223fc837629" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42233 } } }
I20240507 02:11:13.458583 20483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 034af77601684d0388c1f223fc837629 [sys.catalog]: This master's current role is: LEADER
I20240507 02:11:13.458477 20485 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 02:11:13.459113 20485 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240507 02:11:13.460212 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 20410
I20240507 02:11:13.460331 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1715047604443657-32269-0/minicluster-data/master-0/wal/instance
I20240507 02:11:13.460505 32269 external_mini_cluster.cc:1428] Setting key 196b8c186253a83e8792165eb73f4f21
I20240507 02:11:13.460536 20485 catalog_manager.cc:1347] Generated new cluster ID: 6b8c77a5e3f048fc977c7cbd39c9d241
I20240507 02:11:13.460573 20485 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
May 07 02:11:13 dist-test-slave-v0bg krb5kdc[20397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20240507 02:11:13.474361 20501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:13.460944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33333 (local address 127.0.0.1:42233)
0507 02:11:13.461093 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:13.461103 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:13.465295 (+ 4192us) server_negotiation.cc:407] Connection header received
0507 02:11:13.465427 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:13.465433 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:13.465488 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:13.465632 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:13.466523 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:13.467893 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:13.468665 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:13.468890 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:13.471222 (+ 2332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:13.471229 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:13.471229 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:11:13.471246 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:13.473084 (+ 1838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:13.473406 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:13.473409 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:13.473409 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:13.473463 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:13.473596 (+ 133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:13.473596 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:13.473597 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:13.473818 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:13.473943 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:13.474056 (+ 113us) server_negotiation.cc:299] Negotiation successful
0507 02:11:13.474169 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":69,"threads_started":1}
I20240507 02:11:13.475337 32269 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240507 02:11:13.482237 20501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:13.476573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41277 (local address 127.0.0.1:42233)
0507 02:11:13.476604 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:13.476606 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:13.476683 (+ 77us) server_negotiation.cc:407] Connection header received
0507 02:11:13.476725 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:13.476725 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:13.476736 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:13.476799 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:13.477439 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:13.478141 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:13.478796 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:13.478973 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:13.479952 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:13.479955 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:13.479956 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:11:13.479961 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:13.481390 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:13.481688 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:13.481689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:13.481689 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:13.481710 (+ 21us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:13.481824 (+ 114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:13.481825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:13.481825 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:13.481916 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:13.481998 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:13.482077 (+ 79us) server_negotiation.cc:299] Negotiation successful
0507 02:11:13.482149 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20240507 02:11:13.485738 20485 catalog_manager.cc:1370] Generated new certificate authority record
I20240507 02:11:13.486219 20485 catalog_manager.cc:1504] Loading token signing keys...
I20240507 02:11:13.491603 20485 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 034af77601684d0388c1f223fc837629: Generated new TSK 0
I20240507 02:11:13.491752 20485 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240507 02:11:13.493898 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 20410
2024-05-07T02:11:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1144 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1715047604443657-32269-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 07 02:11:13 dist-test-slave-v0bg krb5kdc[20509](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 07 02:11:13 dist-test-slave-v0bg krb5kdc[20509](info): set up 2 sockets
May 07 02:11:13 dist-test-slave-v0bg krb5kdc[20509](info): commencing operation
krb5kdc: starting...
W20240507 02:11:14.371083 32269 mini_kdc.cc:121] Time spent starting KDC: real 0.860s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1715047604443657-32269-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1715047604443657-32269-0/krb5kdc/test-user.keytab.
May 07 02:11:14 dist-test-slave-v0bg krb5kdc[20509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-07T02:11:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-07T02:11:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1715047604443657-32269-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1715047604443657-32269-0/krb5kdc/kudu_127.0.0.1.keytab.
I20240507 02:11:14.467346 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1715047604443657-32269-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1715047604443657-32269-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1715047604443657-32269-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34993
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1715047604443657-32269-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34993
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:11:14.540189 20522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:11:14.540351 20522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:11:14.540380 20522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:11:14.541589 20522 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240507 02:11:14.541625 20522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:11:14.541643 20522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:11:14.541657 20522 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240507 02:11:14.541671 20522 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240507 02:11:14.543576 20522 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1715047604443657-32269-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1715047604443657-32269-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34993
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1715047604443657-32269-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20522
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1715047604443657-32269-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:11:14.543771 20522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:11:14.543958 20522 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:11:14.547691 20527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:11:14.547729 20528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:14.547780 20522 server_base.cc:1034] running on GCE node
W20240507 02:11:14.547866 20530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:14.548094 20522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:11:14.548301 20522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:11:14.549422 20522 hybrid_clock.cc:648] HybridClock initialized: now 1715047874549407 us; error 33 us; skew 500 ppm
May 07 02:11:14 dist-test-slave-v0bg krb5kdc[20509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047874, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:11:14.551532 20522 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20240507 02:11:14.551892 20522 webserver.cc:460] Webserver started at http://127.0.0.1:42745/ using document root <none> and password file <none>
I20240507 02:11:14.552038 20522 fs_manager.cc:359] Metadata directory not provided
I20240507 02:11:14.552076 20522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:11:14.552170 20522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:11:14.553014 20522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1715047604443657-32269-0/minicluster-data/master-0/data/instance:
uuid: "7e996117aa8642e2929ac29ab2ae2c5e"
format_stamp: "Formatted at 2024-05-07 02:11:14 on dist-test-slave-v0bg"
server_key: "1be57e0e3e115c8b572eb9aeedce4c65"
server_key_iv: "701ffb395ae4d381bc84d3f8899f4900"
server_key_version: "encryptionkey@0"
I20240507 02:11:14.553270 20522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1715047604443657-32269-0/minicluster-data/master-0/wal/instance:
uuid: "7e996117aa8642e2929ac29ab2ae2c5e"
format_stamp: "Formatted at 2024-05-07 02:11:14 on dist-test-slave-v0bg"
server_key: "1be57e0e3e115c8b572eb9aeedce4c65"
server_key_iv: "701ffb395ae4d381bc84d3f8899f4900"
server_key_version: "encryptionkey@0"
I20240507 02:11:14.554270 20522 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240507 02:11:14.554872 20537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:14.555009 20522 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:14.555073 20522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1715047604443657-32269-0/minicluster-data/master-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1715047604443657-32269-0/minicluster-data/master-0/wal
uuid: "7e996117aa8642e2929ac29ab2ae2c5e"
format_stamp: "Formatted at 2024-05-07 02:11:14 on dist-test-slave-v0bg"
server_key: "1be57e0e3e115c8b572eb9aeedce4c65"
server_key_iv: "701ffb395ae4d381bc84d3f8899f4900"
server_key_version: "encryptionkey@0"
I20240507 02:11:14.555127 20522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1715047604443657-32269-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1715047604443657-32269-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1715047604443657-32269-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:11:14.564742 20522 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:11:14.566084 20522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:11:14.566179 20522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:11:14.569315 20522 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34993
I20240507 02:11:14.569341 20589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34993 every 8 connection(s)
I20240507 02:11:14.569590 20522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
I20240507 02:11:14.570052 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 20522
I20240507 02:11:14.570135 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1715047604443657-32269-0/minicluster-data/master-0/wal/instance
I20240507 02:11:14.570237 20590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:11:14.570279 32269 external_mini_cluster.cc:1428] Setting key 31cf5424143b76a17d049384c7e4664f
I20240507 02:11:14.572736 20590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e996117aa8642e2929ac29ab2ae2c5e: Bootstrap starting.
I20240507 02:11:14.573300 20590 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e996117aa8642e2929ac29ab2ae2c5e: Neither blocks nor log segments found. Creating new log.
I20240507 02:11:14.573585 20590 log.cc:824] T 00000000000000000000000000000000 P 7e996117aa8642e2929ac29ab2ae2c5e: Log is configured to *not* fsync() on all Append() calls
I20240507 02:11:14.574306 20590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e996117aa8642e2929ac29ab2ae2c5e: No bootstrap required, opened a new log
I20240507 02:11:14.574486 20593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:14.570696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52785 (local address 127.0.0.1:34993)
0507 02:11:14.570820 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:14.570829 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:14.570848 (+ 19us) server_negotiation.cc:407] Connection header received
0507 02:11:14.571453 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:14.571459 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:14.571496 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:14.571603 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:14.572019 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:14.572684 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:14.573415 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:14.573575 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:14.573968 (+ 393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:14.573976 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:14.573976 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0507 02:11:14.574144 (+ 168us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:14.574161 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0507 02:11:14.574168 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:14.574220 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:14.574320 (+ 100us) server_negotiation.cc:299] Negotiation successful
0507 02:11:14.574355 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":60,"threads_started":1}
I20240507 02:11:14.575309 32269 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240507 02:11:14.576020 20590 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e996117aa8642e2929ac29ab2ae2c5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e996117aa8642e2929ac29ab2ae2c5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34993 } }
I20240507 02:11:14.576157 20590 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e996117aa8642e2929ac29ab2ae2c5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:11:14.576185 20590 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e996117aa8642e2929ac29ab2ae2c5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e996117aa8642e2929ac29ab2ae2c5e, State: Initialized, Role: FOLLOWER
I20240507 02:11:14.576277 20590 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e996117aa8642e2929ac29ab2ae2c5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e996117aa8642e2929ac29ab2ae2c5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34993 } }
I20240507 02:11:14.576347 20590 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e996117aa8642e2929ac29ab2ae2c5e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 02:11:14.576376 20590 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e996117aa8642e2929ac29ab2ae2c5e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 02:11:14.576474 20590 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e996117aa8642e2929ac29ab2ae2c5e [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:11:14.577052 20590 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e996117aa8642e2929ac29ab2ae2c5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e996117aa8642e2929ac29ab2ae2c5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34993 } }
I20240507 02:11:14.577149 20590 leader_election.cc:302] T 00000000000000000000000000000000 P 7e996117aa8642e2929ac29ab2ae2c5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e996117aa8642e2929ac29ab2ae2c5e; no voters:
I20240507 02:11:14.577279 20590 leader_election.cc:288] T 00000000000000000000000000000000 P 7e996117aa8642e2929ac29ab2ae2c5e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240507 02:11:14.577323 20600 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e996117aa8642e2929ac29ab2ae2c5e [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:11:14.577462 20590 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e996117aa8642e2929ac29ab2ae2c5e [sys.catalog]: configured and running, proceeding with master startup.
I20240507 02:11:14.577468 20600 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e996117aa8642e2929ac29ab2ae2c5e [term 1 LEADER]: Becoming Leader. State: Replica: 7e996117aa8642e2929ac29ab2ae2c5e, State: Running, Role: LEADER
I20240507 02:11:14.577647 20600 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e996117aa8642e2929ac29ab2ae2c5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e996117aa8642e2929ac29ab2ae2c5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34993 } }
I20240507 02:11:14.577952 20602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e996117aa8642e2929ac29ab2ae2c5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e996117aa8642e2929ac29ab2ae2c5e. Latest consensus state: current_term: 1 leader_uuid: "7e996117aa8642e2929ac29ab2ae2c5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e996117aa8642e2929ac29ab2ae2c5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34993 } } }
I20240507 02:11:14.577946 20601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e996117aa8642e2929ac29ab2ae2c5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e996117aa8642e2929ac29ab2ae2c5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e996117aa8642e2929ac29ab2ae2c5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34993 } } }
I20240507 02:11:14.578038 20602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e996117aa8642e2929ac29ab2ae2c5e [sys.catalog]: This master's current role is: LEADER
I20240507 02:11:14.578066 20601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e996117aa8642e2929ac29ab2ae2c5e [sys.catalog]: This master's current role is: LEADER
I20240507 02:11:14.578217 20608 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 02:11:14.578639 20608 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240507 02:11:14.581000 20608 catalog_manager.cc:1347] Generated new cluster ID: 102e017b903e488d95dd031598587a34
I20240507 02:11:14.581043 20608 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240507 02:11:14.581418 20593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:14.576435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39439 (local address 127.0.0.1:34993)
0507 02:11:14.576477 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:14.576478 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:14.578963 (+ 2485us) server_negotiation.cc:407] Connection header received
0507 02:11:14.579026 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:14.579027 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:14.579035 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:14.579077 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:14.579380 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:14.580045 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:14.580529 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:14.580694 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:14.581056 (+ 362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:14.581059 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:14.581059 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0507 02:11:14.581063 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:14.581075 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0507 02:11:14.581078 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:14.581310 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:14.581326 (+ 16us) server_negotiation.cc:299] Negotiation successful
0507 02:11:14.581350 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20240507 02:11:14.592442 20608 catalog_manager.cc:1370] Generated new certificate authority record
I20240507 02:11:14.592875 20608 catalog_manager.cc:1504] Loading token signing keys...
I20240507 02:11:14.599053 20608 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 7e996117aa8642e2929ac29ab2ae2c5e: Generated new TSK 0
I20240507 02:11:14.599189 20608 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240507 02:11:14.605100 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 20522
2024-05-07T02:11:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1110 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1715047604443657-32269-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 07 02:11:14 dist-test-slave-v0bg krb5kdc[20621](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 07 02:11:14 dist-test-slave-v0bg krb5kdc[20621](info): set up 2 sockets
May 07 02:11:14 dist-test-slave-v0bg krb5kdc[20621](info): commencing operation
krb5kdc: starting...
W20240507 02:11:15.491184 32269 mini_kdc.cc:121] Time spent starting KDC: real 0.869s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1715047604443657-32269-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1715047604443657-32269-0/krb5kdc/test-user.keytab.
May 07 02:11:15 dist-test-slave-v0bg krb5kdc[20621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-07T02:11:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-07T02:11:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1715047604443657-32269-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1715047604443657-32269-0/krb5kdc/kudu_127.0.0.1.keytab.
I20240507 02:11:15.591104 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1715047604443657-32269-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1715047604443657-32269-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1715047604443657-32269-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44215
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1715047604443657-32269-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44215
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:11:15.664544 20634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:11:15.664705 20634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:11:15.664726 20634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:11:15.665953 20634 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240507 02:11:15.665994 20634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:11:15.666007 20634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:11:15.666018 20634 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240507 02:11:15.666047 20634 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240507 02:11:15.667611 20634 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1715047604443657-32269-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1715047604443657-32269-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44215
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1715047604443657-32269-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1715047604443657-32269-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:11:15.667795 20634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:11:15.667953 20634 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:11:15.671414 20640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:11:15.671432 20639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:11:15.671516 20642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:15.671628 20634 server_base.cc:1034] running on GCE node
I20240507 02:11:15.671788 20634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:11:15.671955 20634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:11:15.673084 20634 hybrid_clock.cc:648] HybridClock initialized: now 1715047875673071 us; error 29 us; skew 500 ppm
May 07 02:11:15 dist-test-slave-v0bg krb5kdc[20621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:11:15.675117 20634 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20240507 02:11:15.675438 20634 webserver.cc:460] Webserver started at http://127.0.0.1:38777/ using document root <none> and password file <none>
I20240507 02:11:15.675593 20634 fs_manager.cc:359] Metadata directory not provided
I20240507 02:11:15.675630 20634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:11:15.675721 20634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:11:15.676573 20634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1715047604443657-32269-0/minicluster-data/master-0/data/instance:
uuid: "69e1317bf754481fa2437aa5e6059e1d"
format_stamp: "Formatted at 2024-05-07 02:11:15 on dist-test-slave-v0bg"
server_key: "a996d87fca304383f39712029ff0ac67"
server_key_iv: "c9e32a1028b48d352f5828b347d00a9d"
server_key_version: "encryptionkey@0"
I20240507 02:11:15.676826 20634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1715047604443657-32269-0/minicluster-data/master-0/wal/instance:
uuid: "69e1317bf754481fa2437aa5e6059e1d"
format_stamp: "Formatted at 2024-05-07 02:11:15 on dist-test-slave-v0bg"
server_key: "a996d87fca304383f39712029ff0ac67"
server_key_iv: "c9e32a1028b48d352f5828b347d00a9d"
server_key_version: "encryptionkey@0"
I20240507 02:11:15.677832 20634 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240507 02:11:15.678368 20649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:15.678505 20634 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:15.678596 20634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1715047604443657-32269-0/minicluster-data/master-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1715047604443657-32269-0/minicluster-data/master-0/wal
uuid: "69e1317bf754481fa2437aa5e6059e1d"
format_stamp: "Formatted at 2024-05-07 02:11:15 on dist-test-slave-v0bg"
server_key: "a996d87fca304383f39712029ff0ac67"
server_key_iv: "c9e32a1028b48d352f5828b347d00a9d"
server_key_version: "encryptionkey@0"
I20240507 02:11:15.678655 20634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1715047604443657-32269-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1715047604443657-32269-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1715047604443657-32269-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:11:15.681648 20634 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:11:15.683136 20634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:11:15.683233 20634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:11:15.686185 20634 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44215
I20240507 02:11:15.686213 20701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44215 every 8 connection(s)
I20240507 02:11:15.686439 20634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
I20240507 02:11:15.687184 20702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:11:15.688922 20702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69e1317bf754481fa2437aa5e6059e1d: Bootstrap starting.
I20240507 02:11:15.689384 20702 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69e1317bf754481fa2437aa5e6059e1d: Neither blocks nor log segments found. Creating new log.
I20240507 02:11:15.689581 20702 log.cc:824] T 00000000000000000000000000000000 P 69e1317bf754481fa2437aa5e6059e1d: Log is configured to *not* fsync() on all Append() calls
I20240507 02:11:15.690105 20702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69e1317bf754481fa2437aa5e6059e1d: No bootstrap required, opened a new log
I20240507 02:11:15.691422 20702 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69e1317bf754481fa2437aa5e6059e1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69e1317bf754481fa2437aa5e6059e1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44215 } }
I20240507 02:11:15.691525 20702 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69e1317bf754481fa2437aa5e6059e1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:11:15.691540 20702 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69e1317bf754481fa2437aa5e6059e1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69e1317bf754481fa2437aa5e6059e1d, State: Initialized, Role: FOLLOWER
I20240507 02:11:15.691635 20702 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69e1317bf754481fa2437aa5e6059e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69e1317bf754481fa2437aa5e6059e1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44215 } }
I20240507 02:11:15.691699 20702 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69e1317bf754481fa2437aa5e6059e1d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 02:11:15.691727 20702 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69e1317bf754481fa2437aa5e6059e1d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 02:11:15.691753 20702 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69e1317bf754481fa2437aa5e6059e1d [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:11:15.692189 20702 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69e1317bf754481fa2437aa5e6059e1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69e1317bf754481fa2437aa5e6059e1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44215 } }
I20240507 02:11:15.692262 20702 leader_election.cc:302] T 00000000000000000000000000000000 P 69e1317bf754481fa2437aa5e6059e1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69e1317bf754481fa2437aa5e6059e1d; no voters:
I20240507 02:11:15.692364 20702 leader_election.cc:288] T 00000000000000000000000000000000 P 69e1317bf754481fa2437aa5e6059e1d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240507 02:11:15.692420 20705 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69e1317bf754481fa2437aa5e6059e1d [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:11:15.692552 20702 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69e1317bf754481fa2437aa5e6059e1d [sys.catalog]: configured and running, proceeding with master startup.
I20240507 02:11:15.692579 20705 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69e1317bf754481fa2437aa5e6059e1d [term 1 LEADER]: Becoming Leader. State: Replica: 69e1317bf754481fa2437aa5e6059e1d, State: Running, Role: LEADER
I20240507 02:11:15.692757 20705 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69e1317bf754481fa2437aa5e6059e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69e1317bf754481fa2437aa5e6059e1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44215 } }
I20240507 02:11:15.693066 20706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69e1317bf754481fa2437aa5e6059e1d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69e1317bf754481fa2437aa5e6059e1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69e1317bf754481fa2437aa5e6059e1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44215 } } }
I20240507 02:11:15.693153 20706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69e1317bf754481fa2437aa5e6059e1d [sys.catalog]: This master's current role is: LEADER
I20240507 02:11:15.693138 20707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69e1317bf754481fa2437aa5e6059e1d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69e1317bf754481fa2437aa5e6059e1d. Latest consensus state: current_term: 1 leader_uuid: "69e1317bf754481fa2437aa5e6059e1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69e1317bf754481fa2437aa5e6059e1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44215 } } }
I20240507 02:11:15.693198 20707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69e1317bf754481fa2437aa5e6059e1d [sys.catalog]: This master's current role is: LEADER
I20240507 02:11:15.693326 20711 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 02:11:15.693761 20711 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240507 02:11:15.694126 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 20634
I20240507 02:11:15.694252 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1715047604443657-32269-0/minicluster-data/master-0/wal/instance
I20240507 02:11:15.694403 32269 external_mini_cluster.cc:1428] Setting key 83bcf255e01a69a9d9bd3828b5da864d
I20240507 02:11:15.695348 20711 catalog_manager.cc:1347] Generated new cluster ID: ca800f1070694508b4bee8410a5ae13a
I20240507 02:11:15.695432 20711 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240507 02:11:15.696645 20725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:15.694999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36999 (local address 127.0.0.1:44215)
0507 02:11:15.695385 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:15.695395 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:15.695415 (+ 20us) server_negotiation.cc:407] Connection header received
0507 02:11:15.695504 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:15.695509 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:15.695536 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:15.695671 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:15.696299 (+ 628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:15.696301 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:15.696302 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0507 02:11:15.696304 (+ 2us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:15.696320 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0507 02:11:15.696327 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:15.696399 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:15.696465 (+ 66us) server_negotiation.cc:299] Negotiation successful
0507 02:11:15.696494 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":64,"threads_started":1}
I20240507 02:11:15.697460 32269 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240507 02:11:15.699884 20725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:15.698808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36477 (local address 127.0.0.1:44215)
0507 02:11:15.698838 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:15.698840 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:15.699035 (+ 195us) server_negotiation.cc:407] Connection header received
0507 02:11:15.699058 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:15.699059 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:15.699064 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:15.699119 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:15.699687 (+ 568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:15.699688 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:15.699688 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0507 02:11:15.699693 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:15.699707 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0507 02:11:15.699711 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:15.699773 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:15.699813 (+ 40us) server_negotiation.cc:299] Negotiation successful
0507 02:11:15.699825 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20240507 02:11:15.702631 20711 catalog_manager.cc:1370] Generated new certificate authority record
I20240507 02:11:15.702741 20711 catalog_manager.cc:1504] Loading token signing keys...
I20240507 02:11:15.708117 20711 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 69e1317bf754481fa2437aa5e6059e1d: Generated new TSK 0
I20240507 02:11:15.708245 20711 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240507 02:11:15.713590 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 20634
2024-05-07T02:11:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1108 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1715047604443657-32269-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 07 02:11:15 dist-test-slave-v0bg krb5kdc[20734](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 07 02:11:15 dist-test-slave-v0bg krb5kdc[20734](info): set up 2 sockets
May 07 02:11:15 dist-test-slave-v0bg krb5kdc[20734](info): commencing operation
krb5kdc: starting...
W20240507 02:11:16.599285 32269 mini_kdc.cc:121] Time spent starting KDC: real 0.869s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1715047604443657-32269-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1715047604443657-32269-0/krb5kdc/test-user.keytab.
May 07 02:11:16 dist-test-slave-v0bg krb5kdc[20734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-07T02:11:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-07T02:11:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1715047604443657-32269-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1715047604443657-32269-0/krb5kdc/kudu_127.0.0.1.keytab.
I20240507 02:11:16.707273 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1715047604443657-32269-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1715047604443657-32269-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1715047604443657-32269-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37009
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1715047604443657-32269-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37009
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:11:16.785245 20747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:11:16.785441 20747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:11:16.785463 20747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:11:16.786710 20747 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240507 02:11:16.786754 20747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:11:16.786768 20747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:11:16.786780 20747 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240507 02:11:16.786798 20747 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240507 02:11:16.788415 20747 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1715047604443657-32269-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1715047604443657-32269-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37009
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1715047604443657-32269-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1715047604443657-32269-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:11:16.788630 20747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:11:16.788834 20747 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:11:16.792636 20755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:16.792701 20747 server_base.cc:1034] running on GCE node
W20240507 02:11:16.792676 20753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:11:16.792863 20752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:16.793100 20747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:11:16.793294 20747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:11:16.794425 20747 hybrid_clock.cc:648] HybridClock initialized: now 1715047876794407 us; error 31 us; skew 500 ppm
May 07 02:11:16 dist-test-slave-v0bg krb5kdc[20734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047876, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:11:16.801478 20747 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20240507 02:11:16.801873 20747 webserver.cc:460] Webserver started at http://127.0.0.1:37869/ using document root <none> and password file <none>
I20240507 02:11:16.802035 20747 fs_manager.cc:359] Metadata directory not provided
I20240507 02:11:16.802075 20747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:11:16.802174 20747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:11:16.803149 20747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1715047604443657-32269-0/minicluster-data/master-0/data/instance:
uuid: "90796bc489f3413f8438a765910cb317"
format_stamp: "Formatted at 2024-05-07 02:11:16 on dist-test-slave-v0bg"
server_key: "7e3f16ea781afdac66e8d788e301f116"
server_key_iv: "d53d191154f3db6e75a02ad9a5b33f6d"
server_key_version: "encryptionkey@0"
I20240507 02:11:16.803468 20747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1715047604443657-32269-0/minicluster-data/master-0/wal/instance:
uuid: "90796bc489f3413f8438a765910cb317"
format_stamp: "Formatted at 2024-05-07 02:11:16 on dist-test-slave-v0bg"
server_key: "7e3f16ea781afdac66e8d788e301f116"
server_key_iv: "d53d191154f3db6e75a02ad9a5b33f6d"
server_key_version: "encryptionkey@0"
I20240507 02:11:16.804652 20747 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20240507 02:11:16.805423 20762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:16.805595 20747 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240507 02:11:16.805675 20747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1715047604443657-32269-0/minicluster-data/master-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1715047604443657-32269-0/minicluster-data/master-0/wal
uuid: "90796bc489f3413f8438a765910cb317"
format_stamp: "Formatted at 2024-05-07 02:11:16 on dist-test-slave-v0bg"
server_key: "7e3f16ea781afdac66e8d788e301f116"
server_key_iv: "d53d191154f3db6e75a02ad9a5b33f6d"
server_key_version: "encryptionkey@0"
I20240507 02:11:16.805722 20747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1715047604443657-32269-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1715047604443657-32269-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1715047604443657-32269-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:11:16.817428 20747 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:11:16.818841 20747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:11:16.818928 20747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:11:16.821996 20747 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37009
I20240507 02:11:16.822026 20814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37009 every 8 connection(s)
I20240507 02:11:16.822286 20747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
I20240507 02:11:16.822937 20815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:11:16.824826 20815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90796bc489f3413f8438a765910cb317: Bootstrap starting.
I20240507 02:11:16.825309 20815 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90796bc489f3413f8438a765910cb317: Neither blocks nor log segments found. Creating new log.
I20240507 02:11:16.825556 20815 log.cc:824] T 00000000000000000000000000000000 P 90796bc489f3413f8438a765910cb317: Log is configured to *not* fsync() on all Append() calls
I20240507 02:11:16.826104 20815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90796bc489f3413f8438a765910cb317: No bootstrap required, opened a new log
I20240507 02:11:16.827539 20815 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90796bc489f3413f8438a765910cb317 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90796bc489f3413f8438a765910cb317" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37009 } }
I20240507 02:11:16.827657 20815 raft_consensus.cc:374] T 00000000000000000000000000000000 P 90796bc489f3413f8438a765910cb317 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:11:16.827673 20815 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90796bc489f3413f8438a765910cb317 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90796bc489f3413f8438a765910cb317, State: Initialized, Role: FOLLOWER
I20240507 02:11:16.827769 20815 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90796bc489f3413f8438a765910cb317 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90796bc489f3413f8438a765910cb317" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37009 } }
I20240507 02:11:16.827831 20815 raft_consensus.cc:388] T 00000000000000000000000000000000 P 90796bc489f3413f8438a765910cb317 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 02:11:16.827855 20815 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90796bc489f3413f8438a765910cb317 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 02:11:16.827896 20815 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90796bc489f3413f8438a765910cb317 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:11:16.828356 20815 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90796bc489f3413f8438a765910cb317 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90796bc489f3413f8438a765910cb317" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37009 } }
I20240507 02:11:16.828440 20815 leader_election.cc:302] T 00000000000000000000000000000000 P 90796bc489f3413f8438a765910cb317 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90796bc489f3413f8438a765910cb317; no voters:
I20240507 02:11:16.828568 20815 leader_election.cc:288] T 00000000000000000000000000000000 P 90796bc489f3413f8438a765910cb317 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240507 02:11:16.828609 20818 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90796bc489f3413f8438a765910cb317 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:11:16.828763 20818 raft_consensus.cc:686] T 00000000000000000000000000000000 P 90796bc489f3413f8438a765910cb317 [term 1 LEADER]: Becoming Leader. State: Replica: 90796bc489f3413f8438a765910cb317, State: Running, Role: LEADER
I20240507 02:11:16.828908 20818 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90796bc489f3413f8438a765910cb317 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90796bc489f3413f8438a765910cb317" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37009 } }
I20240507 02:11:16.829087 20815 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90796bc489f3413f8438a765910cb317 [sys.catalog]: configured and running, proceeding with master startup.
I20240507 02:11:16.829317 20819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90796bc489f3413f8438a765910cb317 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "90796bc489f3413f8438a765910cb317" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90796bc489f3413f8438a765910cb317" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37009 } } }
I20240507 02:11:16.829386 20819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90796bc489f3413f8438a765910cb317 [sys.catalog]: This master's current role is: LEADER
I20240507 02:11:16.829505 20820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90796bc489f3413f8438a765910cb317 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90796bc489f3413f8438a765910cb317. Latest consensus state: current_term: 1 leader_uuid: "90796bc489f3413f8438a765910cb317" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90796bc489f3413f8438a765910cb317" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37009 } } }
I20240507 02:11:16.829556 20820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90796bc489f3413f8438a765910cb317 [sys.catalog]: This master's current role is: LEADER
W20240507 02:11:16.830667 20834 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 90796bc489f3413f8438a765910cb317: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240507 02:11:16.830708 20834 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240507 02:11:16.830775 20822 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 02:11:16.830967 20822 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240507 02:11:16.831418 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 20747
I20240507 02:11:16.831504 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1715047604443657-32269-0/minicluster-data/master-0/wal/instance
I20240507 02:11:16.831656 32269 external_mini_cluster.cc:1428] Setting key 54153cc05230d7864cc2fda2c92bdb3c
I20240507 02:11:16.832324 20822 catalog_manager.cc:1347] Generated new cluster ID: 255c640846c24e37a98019171533832e
I20240507 02:11:16.832365 20822 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
May 07 02:11:16 dist-test-slave-v0bg krb5kdc[20734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20240507 02:11:16.841481 20838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:16.832026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.109:42303 (local address 127.0.0.1:37009)
0507 02:11:16.832288 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:16.832301 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:16.832322 (+ 21us) server_negotiation.cc:407] Connection header received
0507 02:11:16.832414 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:16.832419 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:16.832466 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:16.833520 (+ 1054us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:16.833634 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:16.834818 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:16.835781 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:16.837023 (+ 1242us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:16.838044 (+ 1021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:16.838050 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:16.838051 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:11:16.838068 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:16.839956 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:16.840335 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:16.840338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:16.840338 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:16.840398 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:16.840574 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:16.840575 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:16.840575 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:16.840803 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:16.840951 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:16.841062 (+ 111us) server_negotiation.cc:299] Negotiation successful
0507 02:11:16.841281 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":67,"threads_started":1}
I20240507 02:11:16.842461 32269 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240507 02:11:16.843801 20822 catalog_manager.cc:1370] Generated new certificate authority record
I20240507 02:11:16.844419 20822 catalog_manager.cc:1504] Loading token signing keys...
I20240507 02:11:16.849956 20838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:16.843852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.109:32915 (local address 127.0.0.1:37009)
0507 02:11:16.843975 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:16.843977 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:16.844049 (+ 72us) server_negotiation.cc:407] Connection header received
0507 02:11:16.844107 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:16.844108 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:16.844118 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:16.844180 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:16.844890 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:16.845600 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:16.846234 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:16.846409 (+ 175us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:16.847450 (+ 1041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:16.847453 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:16.847453 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:11:16.847460 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:16.848957 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:16.849306 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:16.849307 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:16.849307 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:16.849327 (+ 20us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:16.849495 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:16.849496 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:16.849496 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:16.849597 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:16.849695 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:16.849814 (+ 119us) server_negotiation.cc:299] Negotiation successful
0507 02:11:16.849873 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20240507 02:11:16.854847 20822 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 90796bc489f3413f8438a765910cb317: Generated new TSK 0
I20240507 02:11:16.854990 20822 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240507 02:11:16.859982 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 20747
W20240507 02:11:16.863765 20732 connection.cc:535] client connection to 127.0.0.1:37009 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37009): Connection reset by peer (error 104)
2024-05-07T02:11:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1146 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1715047604443657-32269-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 07 02:11:16 dist-test-slave-v0bg krb5kdc[20846](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 07 02:11:16 dist-test-slave-v0bg krb5kdc[20846](info): set up 2 sockets
May 07 02:11:16 dist-test-slave-v0bg krb5kdc[20846](info): commencing operation
krb5kdc: starting...
W20240507 02:11:17.739280 32269 mini_kdc.cc:121] Time spent starting KDC: real 0.862s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1715047604443657-32269-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1715047604443657-32269-0/krb5kdc/test-user.keytab.
May 07 02:11:17 dist-test-slave-v0bg krb5kdc[20846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-07T02:11:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-07T02:11:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1715047604443657-32269-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1715047604443657-32269-0/krb5kdc/kudu_127.0.0.1.keytab.
I20240507 02:11:17.845777 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1715047604443657-32269-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1715047604443657-32269-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1715047604443657-32269-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33619
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1715047604443657-32269-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33619
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:11:17.921936 20859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:11:17.922116 20859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:11:17.922148 20859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:11:17.923444 20859 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240507 02:11:17.923498 20859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:11:17.923521 20859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:11:17.923540 20859 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240507 02:11:17.923558 20859 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240507 02:11:17.925107 20859 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1715047604443657-32269-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1715047604443657-32269-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33619
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1715047604443657-32269-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1715047604443657-32269-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:11:17.925292 20859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:11:17.925465 20859 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:11:17.929149 20867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:11:17.929157 20865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:17.929177 20859 server_base.cc:1034] running on GCE node
W20240507 02:11:17.929298 20864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:17.929612 20859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:11:17.929833 20859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:11:17.930968 20859 hybrid_clock.cc:648] HybridClock initialized: now 1715047877930953 us; error 28 us; skew 500 ppm
May 07 02:11:17 dist-test-slave-v0bg krb5kdc[20846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047877, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:11:17.933068 20859 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20240507 02:11:17.933457 20859 webserver.cc:460] Webserver started at http://127.0.0.1:35123/ using document root <none> and password file <none>
I20240507 02:11:17.933631 20859 fs_manager.cc:359] Metadata directory not provided
I20240507 02:11:17.933674 20859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:11:17.933777 20859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:11:17.934731 20859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1715047604443657-32269-0/minicluster-data/master-0/data/instance:
uuid: "354fd4e8a9ab4b0d839645b7ee6623ea"
format_stamp: "Formatted at 2024-05-07 02:11:17 on dist-test-slave-v0bg"
server_key: "0bc7ebfe856770e201c61400ad74a662"
server_key_iv: "89f794c67d0cab74957cc00bd60386e8"
server_key_version: "encryptionkey@0"
I20240507 02:11:17.935025 20859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1715047604443657-32269-0/minicluster-data/master-0/wal/instance:
uuid: "354fd4e8a9ab4b0d839645b7ee6623ea"
format_stamp: "Formatted at 2024-05-07 02:11:17 on dist-test-slave-v0bg"
server_key: "0bc7ebfe856770e201c61400ad74a662"
server_key_iv: "89f794c67d0cab74957cc00bd60386e8"
server_key_version: "encryptionkey@0"
I20240507 02:11:17.936094 20859 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240507 02:11:17.936699 20874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:17.936846 20859 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240507 02:11:17.936918 20859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1715047604443657-32269-0/minicluster-data/master-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1715047604443657-32269-0/minicluster-data/master-0/wal
uuid: "354fd4e8a9ab4b0d839645b7ee6623ea"
format_stamp: "Formatted at 2024-05-07 02:11:17 on dist-test-slave-v0bg"
server_key: "0bc7ebfe856770e201c61400ad74a662"
server_key_iv: "89f794c67d0cab74957cc00bd60386e8"
server_key_version: "encryptionkey@0"
I20240507 02:11:17.936978 20859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1715047604443657-32269-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1715047604443657-32269-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1715047604443657-32269-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:11:17.953735 20859 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:11:17.955621 20859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:11:17.955734 20859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:11:17.959237 20859 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33619
I20240507 02:11:17.959265 20926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33619 every 8 connection(s)
I20240507 02:11:17.959563 20859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
I20240507 02:11:17.960208 20927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:11:17.962046 20927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 354fd4e8a9ab4b0d839645b7ee6623ea: Bootstrap starting.
I20240507 02:11:17.962504 20927 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 354fd4e8a9ab4b0d839645b7ee6623ea: Neither blocks nor log segments found. Creating new log.
I20240507 02:11:17.962741 20927 log.cc:824] T 00000000000000000000000000000000 P 354fd4e8a9ab4b0d839645b7ee6623ea: Log is configured to *not* fsync() on all Append() calls
I20240507 02:11:17.963325 20927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 354fd4e8a9ab4b0d839645b7ee6623ea: No bootstrap required, opened a new log
I20240507 02:11:17.964607 20927 raft_consensus.cc:348] T 00000000000000000000000000000000 P 354fd4e8a9ab4b0d839645b7ee6623ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "354fd4e8a9ab4b0d839645b7ee6623ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33619 } }
I20240507 02:11:17.964715 20927 raft_consensus.cc:374] T 00000000000000000000000000000000 P 354fd4e8a9ab4b0d839645b7ee6623ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:11:17.964730 20927 raft_consensus.cc:729] T 00000000000000000000000000000000 P 354fd4e8a9ab4b0d839645b7ee6623ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 354fd4e8a9ab4b0d839645b7ee6623ea, State: Initialized, Role: FOLLOWER
I20240507 02:11:17.964815 20927 consensus_queue.cc:260] T 00000000000000000000000000000000 P 354fd4e8a9ab4b0d839645b7ee6623ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "354fd4e8a9ab4b0d839645b7ee6623ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33619 } }
I20240507 02:11:17.964882 20927 raft_consensus.cc:388] T 00000000000000000000000000000000 P 354fd4e8a9ab4b0d839645b7ee6623ea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 02:11:17.964915 20927 raft_consensus.cc:482] T 00000000000000000000000000000000 P 354fd4e8a9ab4b0d839645b7ee6623ea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 02:11:17.964941 20927 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 354fd4e8a9ab4b0d839645b7ee6623ea [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:11:17.965378 20927 raft_consensus.cc:504] T 00000000000000000000000000000000 P 354fd4e8a9ab4b0d839645b7ee6623ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "354fd4e8a9ab4b0d839645b7ee6623ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33619 } }
I20240507 02:11:17.965456 20927 leader_election.cc:302] T 00000000000000000000000000000000 P 354fd4e8a9ab4b0d839645b7ee6623ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 354fd4e8a9ab4b0d839645b7ee6623ea; no voters:
I20240507 02:11:17.965561 20927 leader_election.cc:288] T 00000000000000000000000000000000 P 354fd4e8a9ab4b0d839645b7ee6623ea [CANDIDATE]: Term 1 election: Requested vote from peers
I20240507 02:11:17.965726 20930 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 354fd4e8a9ab4b0d839645b7ee6623ea [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:11:17.965737 20927 sys_catalog.cc:564] T 00000000000000000000000000000000 P 354fd4e8a9ab4b0d839645b7ee6623ea [sys.catalog]: configured and running, proceeding with master startup.
I20240507 02:11:17.965891 20930 raft_consensus.cc:686] T 00000000000000000000000000000000 P 354fd4e8a9ab4b0d839645b7ee6623ea [term 1 LEADER]: Becoming Leader. State: Replica: 354fd4e8a9ab4b0d839645b7ee6623ea, State: Running, Role: LEADER
I20240507 02:11:17.965986 20930 consensus_queue.cc:237] T 00000000000000000000000000000000 P 354fd4e8a9ab4b0d839645b7ee6623ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "354fd4e8a9ab4b0d839645b7ee6623ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33619 } }
I20240507 02:11:17.966305 20932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 354fd4e8a9ab4b0d839645b7ee6623ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 354fd4e8a9ab4b0d839645b7ee6623ea. Latest consensus state: current_term: 1 leader_uuid: "354fd4e8a9ab4b0d839645b7ee6623ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "354fd4e8a9ab4b0d839645b7ee6623ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33619 } } }
I20240507 02:11:17.966451 20932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 354fd4e8a9ab4b0d839645b7ee6623ea [sys.catalog]: This master's current role is: LEADER
I20240507 02:11:17.966297 20931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 354fd4e8a9ab4b0d839645b7ee6623ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "354fd4e8a9ab4b0d839645b7ee6623ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "354fd4e8a9ab4b0d839645b7ee6623ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33619 } } }
I20240507 02:11:17.966666 20939 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 02:11:17.966706 20931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 354fd4e8a9ab4b0d839645b7ee6623ea [sys.catalog]: This master's current role is: LEADER
I20240507 02:11:17.967103 20939 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240507 02:11:17.968354 20939 catalog_manager.cc:1347] Generated new cluster ID: 390fd5a6fcf440a5b5a515d599af7bb5
I20240507 02:11:17.968407 20939 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240507 02:11:17.969185 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 20859
I20240507 02:11:17.969285 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1715047604443657-32269-0/minicluster-data/master-0/wal/instance
I20240507 02:11:17.969472 32269 external_mini_cluster.cc:1428] Setting key 21edc1d4af4d5ac82bec3e2a875e8c48
I20240507 02:11:17.974082 20939 catalog_manager.cc:1370] Generated new certificate authority record
I20240507 02:11:17.974678 20950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:17.969902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.109:45571 (local address 127.0.0.1:33619)
0507 02:11:17.970065 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:17.970075 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:17.970736 (+ 661us) server_negotiation.cc:407] Connection header received
0507 02:11:17.970858 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:17.970864 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:17.970909 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:17.971055 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:17.971449 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:17.972562 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:17.973350 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:17.973570 (+ 220us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:17.973977 (+ 407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:17.973994 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:17.973994 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0507 02:11:17.974190 (+ 196us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:17.974216 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0507 02:11:17.974225 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:17.974347 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:17.974465 (+ 118us) server_negotiation.cc:299] Negotiation successful
0507 02:11:17.974490 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":78,"threads_started":1}
I20240507 02:11:17.974802 20939 catalog_manager.cc:1504] Loading token signing keys...
I20240507 02:11:17.975656 32269 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240507 02:11:17.979944 20950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:17.977123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.109:57055 (local address 127.0.0.1:33619)
0507 02:11:17.977162 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:17.977164 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:17.977321 (+ 157us) server_negotiation.cc:407] Connection header received
0507 02:11:17.977362 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:17.977363 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:17.977373 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:17.977434 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:17.977732 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:17.978429 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:17.979184 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:17.979333 (+ 149us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:17.979718 (+ 385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:17.979721 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:17.979721 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0507 02:11:17.979725 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:17.979735 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0507 02:11:17.979738 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:17.979788 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:17.979867 (+ 79us) server_negotiation.cc:299] Negotiation successful
0507 02:11:17.979877 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20240507 02:11:17.981407 20939 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 354fd4e8a9ab4b0d839645b7ee6623ea: Generated new TSK 0
I20240507 02:11:17.981556 20939 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240507 02:11:17.986496 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 20859
W20240507 02:11:17.990280 20844 connection.cc:535] client connection to 127.0.0.1:33619 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33619): Connection reset by peer (error 104)
2024-05-07T02:11:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1126 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1715047604443657-32269-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 07 02:11:18 dist-test-slave-v0bg krb5kdc[20958](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 07 02:11:18 dist-test-slave-v0bg krb5kdc[20958](info): set up 2 sockets
May 07 02:11:18 dist-test-slave-v0bg krb5kdc[20958](info): commencing operation
krb5kdc: starting...
W20240507 02:11:18.867265 32269 mini_kdc.cc:121] Time spent starting KDC: real 0.864s user 0.004s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1715047604443657-32269-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1715047604443657-32269-0/krb5kdc/test-user.keytab.
May 07 02:11:18 dist-test-slave-v0bg krb5kdc[20958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047878, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-07T02:11:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-07T02:11:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1715047604443657-32269-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1715047604443657-32269-0/krb5kdc/kudu_127.0.0.1.keytab.
I20240507 02:11:18.965091 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1715047604443657-32269-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1715047604443657-32269-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1715047604443657-32269-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36369
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1715047604443657-32269-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36369
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:11:19.039206 20971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:11:19.039373 20971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:11:19.039392 20971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:11:19.040657 20971 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240507 02:11:19.040700 20971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:11:19.040712 20971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:11:19.040724 20971 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240507 02:11:19.040747 20971 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240507 02:11:19.042270 20971 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1715047604443657-32269-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1715047604443657-32269-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36369
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1715047604443657-32269-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1715047604443657-32269-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:11:19.042448 20971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:11:19.042636 20971 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:11:19.046257 20979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:11:19.046281 20977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:11:19.046416 20976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:19.046422 20971 server_base.cc:1034] running on GCE node
I20240507 02:11:19.046669 20971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:11:19.046850 20971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:11:19.047966 20971 hybrid_clock.cc:648] HybridClock initialized: now 1715047879047956 us; error 29 us; skew 500 ppm
May 07 02:11:19 dist-test-slave-v0bg krb5kdc[20958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047879, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:11:19.050169 20971 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20240507 02:11:19.050565 20971 webserver.cc:460] Webserver started at http://127.0.0.1:46581/ using document root <none> and password file <none>
I20240507 02:11:19.050745 20971 fs_manager.cc:359] Metadata directory not provided
I20240507 02:11:19.050784 20971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:11:19.050894 20971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:11:19.051788 20971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1715047604443657-32269-0/minicluster-data/master-0/data/instance:
uuid: "1e2cdd228e334ff9aeed33c5a298c925"
format_stamp: "Formatted at 2024-05-07 02:11:19 on dist-test-slave-v0bg"
server_key: "e7dd6c20bd7577e8bf89434bc0743fb4"
server_key_iv: "248166f56560a0438852ef0d94c87edc"
server_key_version: "encryptionkey@0"
I20240507 02:11:19.052083 20971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1715047604443657-32269-0/minicluster-data/master-0/wal/instance:
uuid: "1e2cdd228e334ff9aeed33c5a298c925"
format_stamp: "Formatted at 2024-05-07 02:11:19 on dist-test-slave-v0bg"
server_key: "e7dd6c20bd7577e8bf89434bc0743fb4"
server_key_iv: "248166f56560a0438852ef0d94c87edc"
server_key_version: "encryptionkey@0"
I20240507 02:11:19.053149 20971 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20240507 02:11:19.053725 20986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:19.053862 20971 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:19.053936 20971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1715047604443657-32269-0/minicluster-data/master-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1715047604443657-32269-0/minicluster-data/master-0/wal
uuid: "1e2cdd228e334ff9aeed33c5a298c925"
format_stamp: "Formatted at 2024-05-07 02:11:19 on dist-test-slave-v0bg"
server_key: "e7dd6c20bd7577e8bf89434bc0743fb4"
server_key_iv: "248166f56560a0438852ef0d94c87edc"
server_key_version: "encryptionkey@0"
I20240507 02:11:19.054001 20971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1715047604443657-32269-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1715047604443657-32269-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1715047604443657-32269-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:11:19.057451 20971 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:11:19.059198 20971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:11:19.059355 20971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:11:19.062335 20971 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36369
I20240507 02:11:19.062381 21038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36369 every 8 connection(s)
I20240507 02:11:19.062640 20971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
I20240507 02:11:19.063380 21039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:11:19.065274 21039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e2cdd228e334ff9aeed33c5a298c925: Bootstrap starting.
I20240507 02:11:19.065779 21039 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e2cdd228e334ff9aeed33c5a298c925: Neither blocks nor log segments found. Creating new log.
I20240507 02:11:19.066010 21039 log.cc:824] T 00000000000000000000000000000000 P 1e2cdd228e334ff9aeed33c5a298c925: Log is configured to *not* fsync() on all Append() calls
I20240507 02:11:19.066603 21039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e2cdd228e334ff9aeed33c5a298c925: No bootstrap required, opened a new log
I20240507 02:11:19.067934 21039 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e2cdd228e334ff9aeed33c5a298c925 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e2cdd228e334ff9aeed33c5a298c925" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36369 } }
I20240507 02:11:19.068045 21039 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e2cdd228e334ff9aeed33c5a298c925 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:11:19.068064 21039 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e2cdd228e334ff9aeed33c5a298c925 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e2cdd228e334ff9aeed33c5a298c925, State: Initialized, Role: FOLLOWER
I20240507 02:11:19.068167 21039 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e2cdd228e334ff9aeed33c5a298c925 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e2cdd228e334ff9aeed33c5a298c925" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36369 } }
I20240507 02:11:19.068228 21039 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e2cdd228e334ff9aeed33c5a298c925 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 02:11:19.068259 21039 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e2cdd228e334ff9aeed33c5a298c925 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 02:11:19.068310 21039 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e2cdd228e334ff9aeed33c5a298c925 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:11:19.068411 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 20971
I20240507 02:11:19.068516 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1715047604443657-32269-0/minicluster-data/master-0/wal/instance
I20240507 02:11:19.068658 32269 external_mini_cluster.cc:1428] Setting key cdf7460a975f5dc295a36961ea5e159e
I20240507 02:11:19.068922 21039 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e2cdd228e334ff9aeed33c5a298c925 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e2cdd228e334ff9aeed33c5a298c925" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36369 } }
I20240507 02:11:19.069074 21039 leader_election.cc:302] T 00000000000000000000000000000000 P 1e2cdd228e334ff9aeed33c5a298c925 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e2cdd228e334ff9aeed33c5a298c925; no voters:
I20240507 02:11:19.069267 21039 leader_election.cc:288] T 00000000000000000000000000000000 P 1e2cdd228e334ff9aeed33c5a298c925 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240507 02:11:19.069316 21044 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e2cdd228e334ff9aeed33c5a298c925 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:11:19.069484 21044 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e2cdd228e334ff9aeed33c5a298c925 [term 1 LEADER]: Becoming Leader. State: Replica: 1e2cdd228e334ff9aeed33c5a298c925, State: Running, Role: LEADER
I20240507 02:11:19.069504 21039 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e2cdd228e334ff9aeed33c5a298c925 [sys.catalog]: configured and running, proceeding with master startup.
I20240507 02:11:19.069567 21044 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e2cdd228e334ff9aeed33c5a298c925 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e2cdd228e334ff9aeed33c5a298c925" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36369 } }
I20240507 02:11:19.069931 21044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e2cdd228e334ff9aeed33c5a298c925 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e2cdd228e334ff9aeed33c5a298c925. Latest consensus state: current_term: 1 leader_uuid: "1e2cdd228e334ff9aeed33c5a298c925" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e2cdd228e334ff9aeed33c5a298c925" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36369 } } }
I20240507 02:11:19.070071 21044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e2cdd228e334ff9aeed33c5a298c925 [sys.catalog]: This master's current role is: LEADER
I20240507 02:11:19.069938 21045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e2cdd228e334ff9aeed33c5a298c925 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e2cdd228e334ff9aeed33c5a298c925" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e2cdd228e334ff9aeed33c5a298c925" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36369 } } }
I20240507 02:11:19.070294 21048 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 02:11:19.070380 21045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e2cdd228e334ff9aeed33c5a298c925 [sys.catalog]: This master's current role is: LEADER
I20240507 02:11:19.070850 21048 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240507 02:11:19.071146 21043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:19.069034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.109:55241 (local address 127.0.0.1:36369)
0507 02:11:19.069335 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:19.069347 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:19.069367 (+ 20us) server_negotiation.cc:407] Connection header received
0507 02:11:19.069462 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:19.069467 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:19.069495 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:19.069611 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:19.070652 (+ 1041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:19.070655 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:19.070655 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0507 02:11:19.070660 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:19.070683 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0507 02:11:19.070690 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:19.070757 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:19.070946 (+ 189us) server_negotiation.cc:299] Negotiation successful
0507 02:11:19.070979 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":75,"threads_started":1}
I20240507 02:11:19.072075 32269 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240507 02:11:19.074862 21048 catalog_manager.cc:1347] Generated new cluster ID: 3ec0e6e5c76d4b43a1ec732a4c7eddd7
I20240507 02:11:19.074919 21048 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240507 02:11:19.076195 21043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:19.075108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.109:37583 (local address 127.0.0.1:36369)
0507 02:11:19.075148 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:19.075150 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:19.075296 (+ 146us) server_negotiation.cc:407] Connection header received
0507 02:11:19.075361 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:19.075362 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:19.075367 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:19.075413 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:19.075985 (+ 572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:19.075987 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:19.075987 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0507 02:11:19.075998 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:19.076010 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0507 02:11:19.076014 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:19.076055 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:19.076125 (+ 70us) server_negotiation.cc:299] Negotiation successful
0507 02:11:19.076136 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20240507 02:11:19.091361 21048 catalog_manager.cc:1370] Generated new certificate authority record
I20240507 02:11:19.091502 21048 catalog_manager.cc:1504] Loading token signing keys...
I20240507 02:11:19.095579 21048 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 1e2cdd228e334ff9aeed33c5a298c925: Generated new TSK 0
I20240507 02:11:19.095721 21048 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240507 02:11:19.100934 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 20971
2024-05-07T02:11:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1114 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1715047604443657-32269-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 07 02:11:19 dist-test-slave-v0bg krb5kdc[21070](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 07 02:11:19 dist-test-slave-v0bg krb5kdc[21070](info): set up 2 sockets
May 07 02:11:19 dist-test-slave-v0bg krb5kdc[21070](info): commencing operation
krb5kdc: starting...
W20240507 02:11:19.995190 32269 mini_kdc.cc:121] Time spent starting KDC: real 0.877s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1715047604443657-32269-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1715047604443657-32269-0/krb5kdc/test-user.keytab.
May 07 02:11:20 dist-test-slave-v0bg krb5kdc[21070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-07T02:11:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-07T02:11:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1715047604443657-32269-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1715047604443657-32269-0/krb5kdc/kudu_127.0.0.1.keytab.
I20240507 02:11:20.095362 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1715047604443657-32269-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1715047604443657-32269-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1715047604443657-32269-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45057
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1715047604443657-32269-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45057
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:11:20.168689 21083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:11:20.168857 21083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:11:20.168885 21083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:11:20.170114 21083 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240507 02:11:20.170154 21083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:11:20.170172 21083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:11:20.170186 21083 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240507 02:11:20.170200 21083 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240507 02:11:20.171737 21083 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1715047604443657-32269-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1715047604443657-32269-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45057
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1715047604443657-32269-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1715047604443657-32269-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:11:20.171927 21083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:11:20.172087 21083 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240507 02:11:20.175657 21083 server_base.cc:1034] running on GCE node
W20240507 02:11:20.175735 21089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:11:20.175602 21088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:11:20.175581 21091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:20.176023 21083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:11:20.176218 21083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:11:20.177343 21083 hybrid_clock.cc:648] HybridClock initialized: now 1715047880177330 us; error 32 us; skew 500 ppm
May 07 02:11:20 dist-test-slave-v0bg krb5kdc[21070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047880, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:11:20.179579 21083 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20240507 02:11:20.179914 21083 webserver.cc:460] Webserver started at http://127.0.0.1:43645/ using document root <none> and password file <none>
I20240507 02:11:20.180068 21083 fs_manager.cc:359] Metadata directory not provided
I20240507 02:11:20.180106 21083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:11:20.180197 21083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:11:20.181042 21083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1715047604443657-32269-0/minicluster-data/master-0/data/instance:
uuid: "3010f36a00f949c38f4ed39c3b561ed8"
format_stamp: "Formatted at 2024-05-07 02:11:20 on dist-test-slave-v0bg"
server_key: "1c31980066336c1b3c40a8a8e996884f"
server_key_iv: "8d894be04e1ad3eb42f49203d6535d9e"
server_key_version: "encryptionkey@0"
I20240507 02:11:20.181310 21083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1715047604443657-32269-0/minicluster-data/master-0/wal/instance:
uuid: "3010f36a00f949c38f4ed39c3b561ed8"
format_stamp: "Formatted at 2024-05-07 02:11:20 on dist-test-slave-v0bg"
server_key: "1c31980066336c1b3c40a8a8e996884f"
server_key_iv: "8d894be04e1ad3eb42f49203d6535d9e"
server_key_version: "encryptionkey@0"
I20240507 02:11:20.182296 21083 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240507 02:11:20.182876 21098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:20.183012 21083 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:20.183085 21083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1715047604443657-32269-0/minicluster-data/master-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1715047604443657-32269-0/minicluster-data/master-0/wal
uuid: "3010f36a00f949c38f4ed39c3b561ed8"
format_stamp: "Formatted at 2024-05-07 02:11:20 on dist-test-slave-v0bg"
server_key: "1c31980066336c1b3c40a8a8e996884f"
server_key_iv: "8d894be04e1ad3eb42f49203d6535d9e"
server_key_version: "encryptionkey@0"
I20240507 02:11:20.183137 21083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1715047604443657-32269-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1715047604443657-32269-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1715047604443657-32269-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:11:20.196409 21083 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:11:20.198346 21083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:11:20.198458 21083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:11:20.201668 21083 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45057
I20240507 02:11:20.201714 21150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45057 every 8 connection(s)
I20240507 02:11:20.201994 21083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
I20240507 02:11:20.202718 21151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:11:20.204677 21151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3010f36a00f949c38f4ed39c3b561ed8: Bootstrap starting.
I20240507 02:11:20.205199 21151 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3010f36a00f949c38f4ed39c3b561ed8: Neither blocks nor log segments found. Creating new log.
I20240507 02:11:20.205406 21151 log.cc:824] T 00000000000000000000000000000000 P 3010f36a00f949c38f4ed39c3b561ed8: Log is configured to *not* fsync() on all Append() calls
I20240507 02:11:20.205974 21151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3010f36a00f949c38f4ed39c3b561ed8: No bootstrap required, opened a new log
I20240507 02:11:20.207345 21151 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3010f36a00f949c38f4ed39c3b561ed8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3010f36a00f949c38f4ed39c3b561ed8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45057 } }
I20240507 02:11:20.207463 21151 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3010f36a00f949c38f4ed39c3b561ed8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:11:20.207477 21151 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3010f36a00f949c38f4ed39c3b561ed8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3010f36a00f949c38f4ed39c3b561ed8, State: Initialized, Role: FOLLOWER
I20240507 02:11:20.207585 21151 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3010f36a00f949c38f4ed39c3b561ed8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3010f36a00f949c38f4ed39c3b561ed8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45057 } }
I20240507 02:11:20.207654 21151 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3010f36a00f949c38f4ed39c3b561ed8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 02:11:20.207682 21151 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3010f36a00f949c38f4ed39c3b561ed8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 02:11:20.207710 21151 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3010f36a00f949c38f4ed39c3b561ed8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:11:20.208153 21151 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3010f36a00f949c38f4ed39c3b561ed8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3010f36a00f949c38f4ed39c3b561ed8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45057 } }
I20240507 02:11:20.208232 21151 leader_election.cc:302] T 00000000000000000000000000000000 P 3010f36a00f949c38f4ed39c3b561ed8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3010f36a00f949c38f4ed39c3b561ed8; no voters:
I20240507 02:11:20.208386 21151 leader_election.cc:288] T 00000000000000000000000000000000 P 3010f36a00f949c38f4ed39c3b561ed8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240507 02:11:20.208413 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 21083
I20240507 02:11:20.208428 21154 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3010f36a00f949c38f4ed39c3b561ed8 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:11:20.208516 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1715047604443657-32269-0/minicluster-data/master-0/wal/instance
I20240507 02:11:20.208562 21154 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3010f36a00f949c38f4ed39c3b561ed8 [term 1 LEADER]: Becoming Leader. State: Replica: 3010f36a00f949c38f4ed39c3b561ed8, State: Running, Role: LEADER
I20240507 02:11:20.208637 21154 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3010f36a00f949c38f4ed39c3b561ed8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3010f36a00f949c38f4ed39c3b561ed8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45057 } }
I20240507 02:11:20.208674 32269 external_mini_cluster.cc:1428] Setting key 361bb22a4c194631166a8282c3bca265
I20240507 02:11:20.208897 21154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3010f36a00f949c38f4ed39c3b561ed8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3010f36a00f949c38f4ed39c3b561ed8. Latest consensus state: current_term: 1 leader_uuid: "3010f36a00f949c38f4ed39c3b561ed8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3010f36a00f949c38f4ed39c3b561ed8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45057 } } }
I20240507 02:11:20.208946 21151 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3010f36a00f949c38f4ed39c3b561ed8 [sys.catalog]: configured and running, proceeding with master startup.
I20240507 02:11:20.209406 21155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3010f36a00f949c38f4ed39c3b561ed8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3010f36a00f949c38f4ed39c3b561ed8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3010f36a00f949c38f4ed39c3b561ed8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45057 } } }
I20240507 02:11:20.209483 21155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3010f36a00f949c38f4ed39c3b561ed8 [sys.catalog]: This master's current role is: LEADER
I20240507 02:11:20.209745 21164 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 02:11:20.210142 21164 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240507 02:11:20.210347 21154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3010f36a00f949c38f4ed39c3b561ed8 [sys.catalog]: This master's current role is: LEADER
I20240507 02:11:20.211516 21164 catalog_manager.cc:1347] Generated new cluster ID: 366694a23044434986aca604f65b630d
I20240507 02:11:20.211570 21164 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
May 07 02:11:20 dist-test-slave-v0bg krb5kdc[21070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20240507 02:11:20.221118 21159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:20.209272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.109:51011 (local address 127.0.0.1:45057)
0507 02:11:20.209409 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:20.209420 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:20.209441 (+ 21us) server_negotiation.cc:407] Connection header received
0507 02:11:20.209565 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:20.209571 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:20.209623 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:20.209760 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:20.210692 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:20.211661 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:20.212405 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:20.212639 (+ 234us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:20.217043 (+ 4404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:20.217050 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:20.217051 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:11:20.217070 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:20.219645 (+ 2575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:20.220042 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:20.220045 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:20.220045 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:20.220105 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:20.220279 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:20.220280 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:20.220280 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:20.220524 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:20.220688 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:20.220805 (+ 117us) server_negotiation.cc:299] Negotiation successful
0507 02:11:20.220909 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":72,"threads_started":1}
I20240507 02:11:20.222134 32269 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240507 02:11:20.228122 21164 catalog_manager.cc:1370] Generated new certificate authority record
I20240507 02:11:20.228797 21164 catalog_manager.cc:1504] Loading token signing keys...
I20240507 02:11:20.229668 21159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:20.223479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.109:39449 (local address 127.0.0.1:45057)
0507 02:11:20.223560 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:20.223562 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:20.223619 (+ 57us) server_negotiation.cc:407] Connection header received
0507 02:11:20.223676 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:20.223677 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:20.223687 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:20.223746 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:20.224439 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:20.225175 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:20.225811 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:20.225988 (+ 177us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:20.227008 (+ 1020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:20.227011 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:20.227011 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:11:20.227017 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:20.228593 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:20.228970 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:20.228971 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:20.228972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:20.228995 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:20.229168 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:20.229169 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:20.229169 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:20.229252 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:20.229319 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:20.229513 (+ 194us) server_negotiation.cc:299] Negotiation successful
0507 02:11:20.229580 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20240507 02:11:20.235572 21164 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 3010f36a00f949c38f4ed39c3b561ed8: Generated new TSK 0
I20240507 02:11:20.235734 21164 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240507 02:11:20.243988 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 21083
W20240507 02:11:20.247696 21068 connection.cc:535] client connection to 127.0.0.1:45057 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45057): Connection reset by peer (error 104)
2024-05-07T02:11:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1142 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1715047604443657-32269-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 07 02:11:20 dist-test-slave-v0bg krb5kdc[21182](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 07 02:11:20 dist-test-slave-v0bg krb5kdc[21182](info): set up 2 sockets
May 07 02:11:20 dist-test-slave-v0bg krb5kdc[21182](info): commencing operation
krb5kdc: starting...
W20240507 02:11:21.127269 32269 mini_kdc.cc:121] Time spent starting KDC: real 0.867s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1715047604443657-32269-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1715047604443657-32269-0/krb5kdc/test-user.keytab.
May 07 02:11:21 dist-test-slave-v0bg krb5kdc[21182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-07T02:11:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-07T02:11:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1715047604443657-32269-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1715047604443657-32269-0/krb5kdc/kudu_127.0.0.1.keytab.
I20240507 02:11:21.230762 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1715047604443657-32269-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1715047604443657-32269-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1715047604443657-32269-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33677
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1715047604443657-32269-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33677
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:11:21.305109 21195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:11:21.305276 21195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:11:21.305305 21195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:11:21.306562 21195 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240507 02:11:21.306600 21195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:11:21.306617 21195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:11:21.306633 21195 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240507 02:11:21.306646 21195 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240507 02:11:21.308147 21195 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1715047604443657-32269-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1715047604443657-32269-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33677
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1715047604443657-32269-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1715047604443657-32269-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:11:21.308346 21195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:11:21.308516 21195 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:11:21.312021 21201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:11:21.312083 21200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:21.312171 21195 server_base.cc:1034] running on GCE node
W20240507 02:11:21.312249 21203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:21.312470 21195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:11:21.312680 21195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:11:21.313804 21195 hybrid_clock.cc:648] HybridClock initialized: now 1715047881313784 us; error 31 us; skew 500 ppm
May 07 02:11:21 dist-test-slave-v0bg krb5kdc[21182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047881, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:11:21.315968 21195 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20240507 02:11:21.316322 21195 webserver.cc:460] Webserver started at http://127.0.0.1:39271/ using document root <none> and password file <none>
I20240507 02:11:21.316470 21195 fs_manager.cc:359] Metadata directory not provided
I20240507 02:11:21.316504 21195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:11:21.316586 21195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:11:21.317433 21195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1715047604443657-32269-0/minicluster-data/master-0/data/instance:
uuid: "e7c7ed1e35334d73a23e5107013db51a"
format_stamp: "Formatted at 2024-05-07 02:11:21 on dist-test-slave-v0bg"
server_key: "78e1d63cf7c3d09d2b8fb991fbab0cb7"
server_key_iv: "925f8503cf22ab1539da923db81e3db6"
server_key_version: "encryptionkey@0"
I20240507 02:11:21.317683 21195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1715047604443657-32269-0/minicluster-data/master-0/wal/instance:
uuid: "e7c7ed1e35334d73a23e5107013db51a"
format_stamp: "Formatted at 2024-05-07 02:11:21 on dist-test-slave-v0bg"
server_key: "78e1d63cf7c3d09d2b8fb991fbab0cb7"
server_key_iv: "925f8503cf22ab1539da923db81e3db6"
server_key_version: "encryptionkey@0"
I20240507 02:11:21.318665 21195 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240507 02:11:21.319222 21210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:21.319381 21195 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:21.319448 21195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1715047604443657-32269-0/minicluster-data/master-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1715047604443657-32269-0/minicluster-data/master-0/wal
uuid: "e7c7ed1e35334d73a23e5107013db51a"
format_stamp: "Formatted at 2024-05-07 02:11:21 on dist-test-slave-v0bg"
server_key: "78e1d63cf7c3d09d2b8fb991fbab0cb7"
server_key_iv: "925f8503cf22ab1539da923db81e3db6"
server_key_version: "encryptionkey@0"
I20240507 02:11:21.319495 21195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1715047604443657-32269-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1715047604443657-32269-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1715047604443657-32269-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:11:21.327636 21195 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:11:21.329046 21195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:11:21.329130 21195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:11:21.332185 21195 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33677
I20240507 02:11:21.332232 21262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33677 every 8 connection(s)
I20240507 02:11:21.332468 21195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
I20240507 02:11:21.333104 21263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:11:21.333798 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 21195
I20240507 02:11:21.333897 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1715047604443657-32269-0/minicluster-data/master-0/wal/instance
I20240507 02:11:21.334080 32269 external_mini_cluster.cc:1428] Setting key 52cbfc16dde9fab701a593bbd181269d
I20240507 02:11:21.335678 21263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7c7ed1e35334d73a23e5107013db51a: Bootstrap starting.
I20240507 02:11:21.336141 21263 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7c7ed1e35334d73a23e5107013db51a: Neither blocks nor log segments found. Creating new log.
I20240507 02:11:21.336349 21263 log.cc:824] T 00000000000000000000000000000000 P e7c7ed1e35334d73a23e5107013db51a: Log is configured to *not* fsync() on all Append() calls
I20240507 02:11:21.336907 21263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7c7ed1e35334d73a23e5107013db51a: No bootstrap required, opened a new log
I20240507 02:11:21.338017 21266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:21.334520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.109:45667 (local address 127.0.0.1:33677)
0507 02:11:21.334682 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:21.334692 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:21.334712 (+ 20us) server_negotiation.cc:407] Connection header received
0507 02:11:21.335099 (+ 387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:21.335104 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:21.335144 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:21.335259 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:21.335603 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:21.336345 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:21.336903 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:21.337120 (+ 217us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:21.337505 (+ 385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:21.337513 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:21.337514 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0507 02:11:21.337657 (+ 143us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:21.337676 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0507 02:11:21.337682 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:21.337758 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:21.337869 (+ 111us) server_negotiation.cc:299] Negotiation successful
0507 02:11:21.337894 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":91,"threads_started":1}
I20240507 02:11:21.338765 21263 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7c7ed1e35334d73a23e5107013db51a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7c7ed1e35334d73a23e5107013db51a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33677 } }
I20240507 02:11:21.338891 21263 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7c7ed1e35334d73a23e5107013db51a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:11:21.338917 21263 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7c7ed1e35334d73a23e5107013db51a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7c7ed1e35334d73a23e5107013db51a, State: Initialized, Role: FOLLOWER
I20240507 02:11:21.338935 32269 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240507 02:11:21.339002 21263 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7c7ed1e35334d73a23e5107013db51a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7c7ed1e35334d73a23e5107013db51a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33677 } }
I20240507 02:11:21.339056 21263 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7c7ed1e35334d73a23e5107013db51a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 02:11:21.339076 21263 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7c7ed1e35334d73a23e5107013db51a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 02:11:21.339099 21263 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7c7ed1e35334d73a23e5107013db51a [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:11:21.339537 21263 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7c7ed1e35334d73a23e5107013db51a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7c7ed1e35334d73a23e5107013db51a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33677 } }
I20240507 02:11:21.339607 21263 leader_election.cc:302] T 00000000000000000000000000000000 P e7c7ed1e35334d73a23e5107013db51a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7c7ed1e35334d73a23e5107013db51a; no voters:
I20240507 02:11:21.339726 21263 leader_election.cc:288] T 00000000000000000000000000000000 P e7c7ed1e35334d73a23e5107013db51a [CANDIDATE]: Term 1 election: Requested vote from peers
I20240507 02:11:21.339797 21269 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7c7ed1e35334d73a23e5107013db51a [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:11:21.339926 21263 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7c7ed1e35334d73a23e5107013db51a [sys.catalog]: configured and running, proceeding with master startup.
I20240507 02:11:21.339936 21269 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7c7ed1e35334d73a23e5107013db51a [term 1 LEADER]: Becoming Leader. State: Replica: e7c7ed1e35334d73a23e5107013db51a, State: Running, Role: LEADER
I20240507 02:11:21.340090 21269 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7c7ed1e35334d73a23e5107013db51a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7c7ed1e35334d73a23e5107013db51a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33677 } }
I20240507 02:11:21.340384 21269 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7c7ed1e35334d73a23e5107013db51a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7c7ed1e35334d73a23e5107013db51a. Latest consensus state: current_term: 1 leader_uuid: "e7c7ed1e35334d73a23e5107013db51a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7c7ed1e35334d73a23e5107013db51a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33677 } } }
I20240507 02:11:21.340410 21274 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7c7ed1e35334d73a23e5107013db51a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7c7ed1e35334d73a23e5107013db51a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7c7ed1e35334d73a23e5107013db51a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33677 } } }
I20240507 02:11:21.340489 21269 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7c7ed1e35334d73a23e5107013db51a [sys.catalog]: This master's current role is: LEADER
I20240507 02:11:21.340507 21274 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7c7ed1e35334d73a23e5107013db51a [sys.catalog]: This master's current role is: LEADER
I20240507 02:11:21.340755 21281 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 02:11:21.341249 21281 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240507 02:11:21.342655 21281 catalog_manager.cc:1347] Generated new cluster ID: 167d4c05f39a49d3810b0cbc2d2dfb9b
I20240507 02:11:21.342705 21281 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240507 02:11:21.343286 21266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:21.340105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.109:41263 (local address 127.0.0.1:33677)
0507 02:11:21.340143 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:21.340145 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:21.340925 (+ 780us) server_negotiation.cc:407] Connection header received
0507 02:11:21.341001 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:21.341002 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:21.341017 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:21.341061 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:21.341366 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:21.341839 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:21.342457 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:21.342634 (+ 177us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:21.343067 (+ 433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:21.343070 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:21.343071 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0507 02:11:21.343075 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:21.343088 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0507 02:11:21.343092 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:21.343145 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:21.343209 (+ 64us) server_negotiation.cc:299] Negotiation successful
0507 02:11:21.343218 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20240507 02:11:21.360919 21281 catalog_manager.cc:1370] Generated new certificate authority record
I20240507 02:11:21.361451 21281 catalog_manager.cc:1504] Loading token signing keys...
I20240507 02:11:21.366286 21281 catalog_manager.cc:5857] T 00000000000000000000000000000000 P e7c7ed1e35334d73a23e5107013db51a: Generated new TSK 0
I20240507 02:11:21.366448 21281 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240507 02:11:21.373190 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 21195
W20240507 02:11:21.376940 21180 connection.cc:535] client connection to 127.0.0.1:33677 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33677): Connection reset by peer (error 104)
2024-05-07T02:11:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1129 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1715047604443657-32269-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 07 02:11:21 dist-test-slave-v0bg krb5kdc[21294](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 07 02:11:21 dist-test-slave-v0bg krb5kdc[21294](info): set up 2 sockets
May 07 02:11:21 dist-test-slave-v0bg krb5kdc[21294](info): commencing operation
krb5kdc: starting...
W20240507 02:11:22.261114 32269 mini_kdc.cc:121] Time spent starting KDC: real 0.871s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1715047604443657-32269-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1715047604443657-32269-0/krb5kdc/test-user.keytab.
May 07 02:11:22 dist-test-slave-v0bg krb5kdc[21294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047882, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-07T02:11:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-07T02:11:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1715047604443657-32269-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1715047604443657-32269-0/krb5kdc/kudu_127.0.0.1.keytab.
I20240507 02:11:22.361078 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1715047604443657-32269-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1715047604443657-32269-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1715047604443657-32269-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34279
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1715047604443657-32269-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:11:22.434511 21307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:11:22.434688 21307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:11:22.434708 21307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:11:22.435932 21307 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240507 02:11:22.435973 21307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:11:22.435985 21307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:11:22.435998 21307 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240507 02:11:22.436021 21307 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240507 02:11:22.437530 21307 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1715047604443657-32269-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1715047604443657-32269-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34279
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1715047604443657-32269-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1715047604443657-32269-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:11:22.437708 21307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:11:22.437865 21307 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:11:22.441246 21312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:11:22.441287 21313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:11:22.441414 21315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:22.441452 21307 server_base.cc:1034] running on GCE node
I20240507 02:11:22.441632 21307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:11:22.441834 21307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:11:22.442955 21307 hybrid_clock.cc:648] HybridClock initialized: now 1715047882442940 us; error 30 us; skew 500 ppm
May 07 02:11:22 dist-test-slave-v0bg krb5kdc[21294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047882, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:11:22.450269 21307 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20240507 02:11:22.450718 21307 webserver.cc:460] Webserver started at http://127.0.0.1:46605/ using document root <none> and password file <none>
I20240507 02:11:22.450904 21307 fs_manager.cc:359] Metadata directory not provided
I20240507 02:11:22.450946 21307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:11:22.451054 21307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:11:22.451987 21307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1715047604443657-32269-0/minicluster-data/master-0/data/instance:
uuid: "25c8968614304d6cbaa49ccca61d8f95"
format_stamp: "Formatted at 2024-05-07 02:11:22 on dist-test-slave-v0bg"
server_key: "a6bf7218c86b19070fe774a4efea07fe"
server_key_iv: "6926b8e16ac23dec6f5a9b2e25eda98e"
server_key_version: "encryptionkey@0"
I20240507 02:11:22.452265 21307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1715047604443657-32269-0/minicluster-data/master-0/wal/instance:
uuid: "25c8968614304d6cbaa49ccca61d8f95"
format_stamp: "Formatted at 2024-05-07 02:11:22 on dist-test-slave-v0bg"
server_key: "a6bf7218c86b19070fe774a4efea07fe"
server_key_iv: "6926b8e16ac23dec6f5a9b2e25eda98e"
server_key_version: "encryptionkey@0"
I20240507 02:11:22.453315 21307 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240507 02:11:22.453896 21322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:22.454046 21307 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240507 02:11:22.454118 21307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1715047604443657-32269-0/minicluster-data/master-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1715047604443657-32269-0/minicluster-data/master-0/wal
uuid: "25c8968614304d6cbaa49ccca61d8f95"
format_stamp: "Formatted at 2024-05-07 02:11:22 on dist-test-slave-v0bg"
server_key: "a6bf7218c86b19070fe774a4efea07fe"
server_key_iv: "6926b8e16ac23dec6f5a9b2e25eda98e"
server_key_version: "encryptionkey@0"
I20240507 02:11:22.454173 21307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1715047604443657-32269-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1715047604443657-32269-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1715047604443657-32269-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:11:22.457468 21307 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:11:22.458992 21307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:11:22.459096 21307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:11:22.462275 21307 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34279
I20240507 02:11:22.462302 21374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34279 every 8 connection(s)
I20240507 02:11:22.462600 21307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
I20240507 02:11:22.463352 21375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:11:22.464159 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 21307
I20240507 02:11:22.464247 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1715047604443657-32269-0/minicluster-data/master-0/wal/instance
I20240507 02:11:22.464390 32269 external_mini_cluster.cc:1428] Setting key 8c955832e241332d25cd5e8ec5c02dd4
I20240507 02:11:22.465548 21375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25c8968614304d6cbaa49ccca61d8f95: Bootstrap starting.
I20240507 02:11:22.466109 21375 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25c8968614304d6cbaa49ccca61d8f95: Neither blocks nor log segments found. Creating new log.
I20240507 02:11:22.466351 21375 log.cc:824] T 00000000000000000000000000000000 P 25c8968614304d6cbaa49ccca61d8f95: Log is configured to *not* fsync() on all Append() calls
I20240507 02:11:22.466598 21378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:22.464824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.109:43267 (local address 127.0.0.1:34279)
0507 02:11:22.464955 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:22.464963 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:22.464981 (+ 18us) server_negotiation.cc:407] Connection header received
0507 02:11:22.465439 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:22.465444 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:22.465471 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:22.465597 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:22.466226 (+ 629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:22.466229 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:22.466229 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0507 02:11:22.466233 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:22.466256 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0507 02:11:22.466262 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:22.466319 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:22.466389 (+ 70us) server_negotiation.cc:299] Negotiation successful
0507 02:11:22.466420 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":57,"threads_started":1}
I20240507 02:11:22.466982 21375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25c8968614304d6cbaa49ccca61d8f95: No bootstrap required, opened a new log
I20240507 02:11:22.467420 32269 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240507 02:11:22.468685 21375 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25c8968614304d6cbaa49ccca61d8f95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25c8968614304d6cbaa49ccca61d8f95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34279 } }
I20240507 02:11:22.468798 21375 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25c8968614304d6cbaa49ccca61d8f95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:11:22.468813 21375 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25c8968614304d6cbaa49ccca61d8f95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25c8968614304d6cbaa49ccca61d8f95, State: Initialized, Role: FOLLOWER
I20240507 02:11:22.468906 21375 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25c8968614304d6cbaa49ccca61d8f95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25c8968614304d6cbaa49ccca61d8f95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34279 } }
I20240507 02:11:22.468964 21375 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25c8968614304d6cbaa49ccca61d8f95 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 02:11:22.468983 21375 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25c8968614304d6cbaa49ccca61d8f95 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 02:11:22.469023 21375 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25c8968614304d6cbaa49ccca61d8f95 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:11:22.469578 21378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:22.468564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.109:58941 (local address 127.0.0.1:34279)
0507 02:11:22.468594 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:22.468596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:22.468728 (+ 132us) server_negotiation.cc:407] Connection header received
0507 02:11:22.468783 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:22.468783 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:22.468788 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:22.468832 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:22.469400 (+ 568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:22.469401 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:22.469401 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0507 02:11:22.469405 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:22.469417 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0507 02:11:22.469420 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:22.469464 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:22.469508 (+ 44us) server_negotiation.cc:299] Negotiation successful
0507 02:11:22.469521 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20240507 02:11:22.469627 21375 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25c8968614304d6cbaa49ccca61d8f95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25c8968614304d6cbaa49ccca61d8f95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34279 } }
I20240507 02:11:22.469722 21375 leader_election.cc:302] T 00000000000000000000000000000000 P 25c8968614304d6cbaa49ccca61d8f95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25c8968614304d6cbaa49ccca61d8f95; no voters:
I20240507 02:11:22.469866 21375 leader_election.cc:288] T 00000000000000000000000000000000 P 25c8968614304d6cbaa49ccca61d8f95 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240507 02:11:22.469982 21385 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25c8968614304d6cbaa49ccca61d8f95 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:11:22.470088 21375 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25c8968614304d6cbaa49ccca61d8f95 [sys.catalog]: configured and running, proceeding with master startup.
I20240507 02:11:22.470108 21385 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25c8968614304d6cbaa49ccca61d8f95 [term 1 LEADER]: Becoming Leader. State: Replica: 25c8968614304d6cbaa49ccca61d8f95, State: Running, Role: LEADER
I20240507 02:11:22.470187 21385 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25c8968614304d6cbaa49ccca61d8f95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25c8968614304d6cbaa49ccca61d8f95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34279 } }
I20240507 02:11:22.470597 21386 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25c8968614304d6cbaa49ccca61d8f95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25c8968614304d6cbaa49ccca61d8f95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25c8968614304d6cbaa49ccca61d8f95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34279 } } }
I20240507 02:11:22.470679 21386 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25c8968614304d6cbaa49ccca61d8f95 [sys.catalog]: This master's current role is: LEADER
I20240507 02:11:22.470584 21387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25c8968614304d6cbaa49ccca61d8f95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25c8968614304d6cbaa49ccca61d8f95. Latest consensus state: current_term: 1 leader_uuid: "25c8968614304d6cbaa49ccca61d8f95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25c8968614304d6cbaa49ccca61d8f95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34279 } } }
I20240507 02:11:22.470814 21387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25c8968614304d6cbaa49ccca61d8f95 [sys.catalog]: This master's current role is: LEADER
I20240507 02:11:22.470927 21391 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 02:11:22.471359 21391 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240507 02:11:22.472640 21391 catalog_manager.cc:1347] Generated new cluster ID: 0ef3376316ce4afe98efc7c84af8ec63
I20240507 02:11:22.472681 21391 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240507 02:11:22.489346 21391 catalog_manager.cc:1370] Generated new certificate authority record
I20240507 02:11:22.489483 21391 catalog_manager.cc:1504] Loading token signing keys...
I20240507 02:11:22.493197 21391 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 25c8968614304d6cbaa49ccca61d8f95: Generated new TSK 0
I20240507 02:11:22.493386 21391 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240507 02:11:22.498380 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 21307
2024-05-07T02:11:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1125 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1715047604443657-32269-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 07 02:11:22 dist-test-slave-v0bg krb5kdc[21406](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 07 02:11:22 dist-test-slave-v0bg krb5kdc[21406](info): set up 2 sockets
May 07 02:11:22 dist-test-slave-v0bg krb5kdc[21406](info): commencing operation
krb5kdc: starting...
W20240507 02:11:23.379240 32269 mini_kdc.cc:121] Time spent starting KDC: real 0.864s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1715047604443657-32269-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1715047604443657-32269-0/krb5kdc/test-user.keytab.
May 07 02:11:23 dist-test-slave-v0bg krb5kdc[21406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-07T02:11:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-07T02:11:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.131.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab.
Entry for principal kudu/127.31.131.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab.
I20240507 02:11:23.481122 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1715047604443657-32269-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1715047604443657-32269-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1715047604443657-32269-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:42149
--webserver_interface=127.31.131.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab
--principal=kudu/127.31.131.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:41241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.131.126:42149
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:11:23.559361 21419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:11:23.559594 21419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:11:23.559674 21419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:11:23.561650 21419 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240507 02:11:23.561703 21419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:11:23.561720 21419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:11:23.561738 21419 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240507 02:11:23.561751 21419 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240507 02:11:23.563884 21419 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:41241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1715047604443657-32269-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1715047604443657-32269-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.131.126:42149
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab
--principal=kudu/127.31.131.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:42149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.131.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1715047604443657-32269-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:11:23.564105 21419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:11:23.564278 21419 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:11:23.568006 21425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:11:23.568019 21424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:11:23.568043 21427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:23.571457 21419 server_base.cc:1034] running on GCE node
I20240507 02:11:23.571691 21419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:11:23.571933 21419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:11:23.573065 21419 hybrid_clock.cc:648] HybridClock initialized: now 1715047883573052 us; error 31 us; skew 500 ppm
May 07 02:11:23 dist-test-slave-v0bg krb5kdc[21406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047883, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:11:23.575222 21419 init.cc:377] Logged in from keytab as kudu/127.31.131.126@KRBTEST.COM (short username kudu)
I20240507 02:11:23.575593 21419 webserver.cc:460] Webserver started at http://127.31.131.126:36975/ using document root <none> and password file <none>
I20240507 02:11:23.575747 21419 fs_manager.cc:359] Metadata directory not provided
I20240507 02:11:23.575784 21419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:11:23.575879 21419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:11:23.576777 21419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1715047604443657-32269-0/minicluster-data/master-0/data/instance:
uuid: "71fbb235852e4411a2b1d50dc7b843a0"
format_stamp: "Formatted at 2024-05-07 02:11:23 on dist-test-slave-v0bg"
server_key: "8da3f98c1b51dc7f8c8735d8bcf09d06"
server_key_iv: "22673a673219f89722493f1c45450881"
server_key_version: "encryptionkey@0"
I20240507 02:11:23.577054 21419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1715047604443657-32269-0/minicluster-data/master-0/wal/instance:
uuid: "71fbb235852e4411a2b1d50dc7b843a0"
format_stamp: "Formatted at 2024-05-07 02:11:23 on dist-test-slave-v0bg"
server_key: "8da3f98c1b51dc7f8c8735d8bcf09d06"
server_key_iv: "22673a673219f89722493f1c45450881"
server_key_version: "encryptionkey@0"
I20240507 02:11:23.578070 21419 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240507 02:11:23.578649 21434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:23.578776 21419 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:23.578846 21419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1715047604443657-32269-0/minicluster-data/master-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1715047604443657-32269-0/minicluster-data/master-0/wal
uuid: "71fbb235852e4411a2b1d50dc7b843a0"
format_stamp: "Formatted at 2024-05-07 02:11:23 on dist-test-slave-v0bg"
server_key: "8da3f98c1b51dc7f8c8735d8bcf09d06"
server_key_iv: "22673a673219f89722493f1c45450881"
server_key_version: "encryptionkey@0"
I20240507 02:11:23.578904 21419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1715047604443657-32269-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1715047604443657-32269-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1715047604443657-32269-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:11:23.617666 21419 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:11:23.619856 21419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:11:23.619997 21419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:11:23.624480 21419 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.126:42149
I20240507 02:11:23.624526 21486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.126:42149 every 8 connection(s)
I20240507 02:11:23.624898 21419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
I20240507 02:11:23.625255 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 21419
I20240507 02:11:23.625351 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1715047604443657-32269-0/minicluster-data/master-0/wal/instance
I20240507 02:11:23.625520 32269 external_mini_cluster.cc:1428] Setting key a789d3a6317bf655a6ad1ff296dab72c
I20240507 02:11:23.625797 21487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:11:23.628813 21487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71fbb235852e4411a2b1d50dc7b843a0: Bootstrap starting.
I20240507 02:11:23.629439 21487 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71fbb235852e4411a2b1d50dc7b843a0: Neither blocks nor log segments found. Creating new log.
I20240507 02:11:23.629734 21487 log.cc:824] T 00000000000000000000000000000000 P 71fbb235852e4411a2b1d50dc7b843a0: Log is configured to *not* fsync() on all Append() calls
I20240507 02:11:23.630457 21487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71fbb235852e4411a2b1d50dc7b843a0: No bootstrap required, opened a new log
May 07 02:11:23 dist-test-slave-v0bg krb5kdc[21406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:11:23.632531 21487 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71fbb235852e4411a2b1d50dc7b843a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71fbb235852e4411a2b1d50dc7b843a0" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 42149 } }
I20240507 02:11:23.632688 21487 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71fbb235852e4411a2b1d50dc7b843a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:11:23.632714 21487 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71fbb235852e4411a2b1d50dc7b843a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71fbb235852e4411a2b1d50dc7b843a0, State: Initialized, Role: FOLLOWER
I20240507 02:11:23.632836 21487 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71fbb235852e4411a2b1d50dc7b843a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71fbb235852e4411a2b1d50dc7b843a0" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 42149 } }
I20240507 02:11:23.632894 21487 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71fbb235852e4411a2b1d50dc7b843a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 02:11:23.632913 21487 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71fbb235852e4411a2b1d50dc7b843a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 02:11:23.632957 21487 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71fbb235852e4411a2b1d50dc7b843a0 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:11:23.633464 21487 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71fbb235852e4411a2b1d50dc7b843a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71fbb235852e4411a2b1d50dc7b843a0" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 42149 } }
I20240507 02:11:23.633571 21487 leader_election.cc:302] T 00000000000000000000000000000000 P 71fbb235852e4411a2b1d50dc7b843a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71fbb235852e4411a2b1d50dc7b843a0; no voters:
I20240507 02:11:23.633713 21487 leader_election.cc:288] T 00000000000000000000000000000000 P 71fbb235852e4411a2b1d50dc7b843a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240507 02:11:23.633774 21492 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71fbb235852e4411a2b1d50dc7b843a0 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:11:23.633909 21492 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71fbb235852e4411a2b1d50dc7b843a0 [term 1 LEADER]: Becoming Leader. State: Replica: 71fbb235852e4411a2b1d50dc7b843a0, State: Running, Role: LEADER
I20240507 02:11:23.633953 21487 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71fbb235852e4411a2b1d50dc7b843a0 [sys.catalog]: configured and running, proceeding with master startup.
I20240507 02:11:23.634001 21492 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71fbb235852e4411a2b1d50dc7b843a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71fbb235852e4411a2b1d50dc7b843a0" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 42149 } }
I20240507 02:11:23.634325 21494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71fbb235852e4411a2b1d50dc7b843a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71fbb235852e4411a2b1d50dc7b843a0. Latest consensus state: current_term: 1 leader_uuid: "71fbb235852e4411a2b1d50dc7b843a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71fbb235852e4411a2b1d50dc7b843a0" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 42149 } } }
I20240507 02:11:23.634394 21494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71fbb235852e4411a2b1d50dc7b843a0 [sys.catalog]: This master's current role is: LEADER
I20240507 02:11:23.634660 21493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71fbb235852e4411a2b1d50dc7b843a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71fbb235852e4411a2b1d50dc7b843a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71fbb235852e4411a2b1d50dc7b843a0" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 42149 } } }
I20240507 02:11:23.634781 21493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71fbb235852e4411a2b1d50dc7b843a0 [sys.catalog]: This master's current role is: LEADER
I20240507 02:11:23.634795 21497 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 02:11:23.635346 21497 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240507 02:11:23.635967 21490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:23.625989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45413 (local address 127.31.131.126:42149)
0507 02:11:23.626183 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:23.626195 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:23.626217 (+ 22us) server_negotiation.cc:407] Connection header received
0507 02:11:23.626870 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:23.626880 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:23.626937 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:23.627104 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:23.627777 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:23.628915 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:23.629694 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:23.629901 (+ 207us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:23.632195 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:23.632203 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:23.632203 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:11:23.632220 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:23.634127 (+ 1907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:23.634543 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:23.634580 (+ 37us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:23.634580 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:23.634649 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:23.634865 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:23.634866 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:23.634866 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:23.635140 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:23.635330 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:23.635751 (+ 421us) server_negotiation.cc:299] Negotiation successful
0507 02:11:23.635830 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":102,"threads_started":1}
I20240507 02:11:23.636832 32269 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240507 02:11:23.639267 21509 mvcc.cc:205] Tried to move back new op lower bound from 7024836131384553472 to 7024836131365265408. Current Snapshot: MvccSnapshot[applied={T|T < 7024836131384553472}]
I20240507 02:11:23.639680 21497 catalog_manager.cc:1347] Generated new cluster ID: 75faf8f7839a4175ad1446024994ede0
I20240507 02:11:23.639729 21497 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240507 02:11:23.644326 21490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:23.637996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34001 (local address 127.31.131.126:42149)
0507 02:11:23.638036 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:23.638038 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:23.638144 (+ 106us) server_negotiation.cc:407] Connection header received
0507 02:11:23.638203 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:23.638204 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:23.638212 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:23.638256 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:23.638883 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:23.639415 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:23.640329 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:23.640523 (+ 194us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:23.641522 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:23.641526 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:23.641526 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:11:23.641532 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:23.643007 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:23.643398 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:23.643399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:23.643399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:23.643422 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:23.643611 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:23.643611 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:23.643612 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:23.643715 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:23.643807 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:23.644178 (+ 371us) server_negotiation.cc:299] Negotiation successful
0507 02:11:23.644234 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20240507 02:11:23.660950 21497 catalog_manager.cc:1370] Generated new certificate authority record
I20240507 02:11:23.661466 21497 catalog_manager.cc:1504] Loading token signing keys...
I20240507 02:11:23.667357 21497 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 71fbb235852e4411a2b1d50dc7b843a0: Generated new TSK 0
I20240507 02:11:23.667565 21497 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240507 02:11:23.675778 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 21419
W20240507 02:11:23.679541 21404 connection.cc:535] client connection to 127.31.131.126:42149 recv error: Network error: failed to read from TLS socket (remote: 127.31.131.126:42149): Connection reset by peer (error 104)
2024-05-07T02:11:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1177 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1715047604443657-32269-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 07 02:11:23 dist-test-slave-v0bg krb5kdc[21518](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 07 02:11:23 dist-test-slave-v0bg krb5kdc[21518](info): set up 2 sockets
May 07 02:11:23 dist-test-slave-v0bg krb5kdc[21518](info): commencing operation
krb5kdc: starting...
W20240507 02:11:24.559450 32269 mini_kdc.cc:121] Time spent starting KDC: real 0.867s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1715047604443657-32269-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1715047604443657-32269-0/krb5kdc/test-user.keytab.
May 07 02:11:24 dist-test-slave-v0bg krb5kdc[21518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-07T02:11:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-07T02:11:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.131.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab.
Entry for principal kudu/127.31.131.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab.
I20240507 02:11:24.666736 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1715047604443657-32269-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1715047604443657-32269-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1715047604443657-32269-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:39039
--webserver_interface=127.31.131.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab
--principal=kudu/127.31.131.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:44075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.131.126:39039
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:11:24.744967 21531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:11:24.745191 21531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:11:24.745223 21531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:11:24.746614 21531 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240507 02:11:24.746666 21531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:11:24.746685 21531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:11:24.746698 21531 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240507 02:11:24.746712 21531 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240507 02:11:24.748379 21531 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:44075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1715047604443657-32269-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1715047604443657-32269-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.131.126:39039
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab
--principal=kudu/127.31.131.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:39039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.131.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1715047604443657-32269-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:11:24.748615 21531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:11:24.748783 21531 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:11:24.752403 21537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:11:24.752439 21539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:24.752542 21531 server_base.cc:1034] running on GCE node
W20240507 02:11:24.752667 21536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:24.752907 21531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:11:24.753116 21531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:11:24.754251 21531 hybrid_clock.cc:648] HybridClock initialized: now 1715047884754230 us; error 37 us; skew 500 ppm
May 07 02:11:24 dist-test-slave-v0bg krb5kdc[21518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047884, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:11:24.756645 21531 init.cc:377] Logged in from keytab as kudu/127.31.131.126@KRBTEST.COM (short username kudu)
I20240507 02:11:24.757058 21531 webserver.cc:460] Webserver started at http://127.31.131.126:38321/ using document root <none> and password file <none>
I20240507 02:11:24.757228 21531 fs_manager.cc:359] Metadata directory not provided
I20240507 02:11:24.757268 21531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:11:24.757371 21531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:11:24.758319 21531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1715047604443657-32269-0/minicluster-data/master-0/data/instance:
uuid: "61eae55814274a1db8dc6dd0fb52a942"
format_stamp: "Formatted at 2024-05-07 02:11:24 on dist-test-slave-v0bg"
server_key: "107616404b3422773a10ce65d97250db"
server_key_iv: "30dd2409f72382d67d99f011a42131e7"
server_key_version: "encryptionkey@0"
I20240507 02:11:24.758657 21531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1715047604443657-32269-0/minicluster-data/master-0/wal/instance:
uuid: "61eae55814274a1db8dc6dd0fb52a942"
format_stamp: "Formatted at 2024-05-07 02:11:24 on dist-test-slave-v0bg"
server_key: "107616404b3422773a10ce65d97250db"
server_key_iv: "30dd2409f72382d67d99f011a42131e7"
server_key_version: "encryptionkey@0"
I20240507 02:11:24.759750 21531 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240507 02:11:24.760363 21546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:24.760524 21531 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240507 02:11:24.760602 21531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1715047604443657-32269-0/minicluster-data/master-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1715047604443657-32269-0/minicluster-data/master-0/wal
uuid: "61eae55814274a1db8dc6dd0fb52a942"
format_stamp: "Formatted at 2024-05-07 02:11:24 on dist-test-slave-v0bg"
server_key: "107616404b3422773a10ce65d97250db"
server_key_iv: "30dd2409f72382d67d99f011a42131e7"
server_key_version: "encryptionkey@0"
I20240507 02:11:24.760656 21531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1715047604443657-32269-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1715047604443657-32269-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1715047604443657-32269-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:11:24.772429 21531 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:11:24.774400 21531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:11:24.774518 21531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:11:24.777832 21531 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.126:39039
I20240507 02:11:24.777871 21598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.126:39039 every 8 connection(s)
I20240507 02:11:24.778172 21531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
I20240507 02:11:24.778898 21599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:11:24.780352 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 21531
I20240507 02:11:24.780460 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1715047604443657-32269-0/minicluster-data/master-0/wal/instance
I20240507 02:11:24.780651 32269 external_mini_cluster.cc:1428] Setting key 3a5c3c6a611e085d103ae44ff3587af1
I20240507 02:11:24.781190 21599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61eae55814274a1db8dc6dd0fb52a942: Bootstrap starting.
I20240507 02:11:24.781831 21599 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61eae55814274a1db8dc6dd0fb52a942: Neither blocks nor log segments found. Creating new log.
I20240507 02:11:24.782122 21599 log.cc:824] T 00000000000000000000000000000000 P 61eae55814274a1db8dc6dd0fb52a942: Log is configured to *not* fsync() on all Append() calls
I20240507 02:11:24.782852 21599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61eae55814274a1db8dc6dd0fb52a942: No bootstrap required, opened a new log
I20240507 02:11:24.784776 21599 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61eae55814274a1db8dc6dd0fb52a942 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61eae55814274a1db8dc6dd0fb52a942" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 39039 } }
I20240507 02:11:24.784930 21599 raft_consensus.cc:374] T 00000000000000000000000000000000 P 61eae55814274a1db8dc6dd0fb52a942 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:11:24.784955 21599 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61eae55814274a1db8dc6dd0fb52a942 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61eae55814274a1db8dc6dd0fb52a942, State: Initialized, Role: FOLLOWER
I20240507 02:11:24.785071 21599 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61eae55814274a1db8dc6dd0fb52a942 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61eae55814274a1db8dc6dd0fb52a942" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 39039 } }
I20240507 02:11:24.785149 21599 raft_consensus.cc:388] T 00000000000000000000000000000000 P 61eae55814274a1db8dc6dd0fb52a942 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 02:11:24.785187 21599 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61eae55814274a1db8dc6dd0fb52a942 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 02:11:24.785221 21599 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61eae55814274a1db8dc6dd0fb52a942 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:11:24.785418 21602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:24.781174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52677 (local address 127.31.131.126:39039)
0507 02:11:24.781358 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:24.781370 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:24.781393 (+ 23us) server_negotiation.cc:407] Connection header received
0507 02:11:24.781959 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:24.781971 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:24.782034 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:24.782204 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:24.782586 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:24.783691 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:24.784270 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:24.784547 (+ 277us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:24.784832 (+ 285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:24.784839 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:24.784839 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0507 02:11:24.785015 (+ 176us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:24.785042 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0507 02:11:24.785051 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:24.785145 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:24.785226 (+ 81us) server_negotiation.cc:299] Negotiation successful
0507 02:11:24.785252 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":78,"threads_started":1}
I20240507 02:11:24.785832 21599 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61eae55814274a1db8dc6dd0fb52a942 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61eae55814274a1db8dc6dd0fb52a942" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 39039 } }
I20240507 02:11:24.785929 21599 leader_election.cc:302] T 00000000000000000000000000000000 P 61eae55814274a1db8dc6dd0fb52a942 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61eae55814274a1db8dc6dd0fb52a942; no voters:
I20240507 02:11:24.786103 21599 leader_election.cc:288] T 00000000000000000000000000000000 P 61eae55814274a1db8dc6dd0fb52a942 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240507 02:11:24.786196 21604 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61eae55814274a1db8dc6dd0fb52a942 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:11:24.786356 21599 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61eae55814274a1db8dc6dd0fb52a942 [sys.catalog]: configured and running, proceeding with master startup.
I20240507 02:11:24.786386 32269 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240507 02:11:24.786382 21604 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61eae55814274a1db8dc6dd0fb52a942 [term 1 LEADER]: Becoming Leader. State: Replica: 61eae55814274a1db8dc6dd0fb52a942, State: Running, Role: LEADER
I20240507 02:11:24.786474 21604 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61eae55814274a1db8dc6dd0fb52a942 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61eae55814274a1db8dc6dd0fb52a942" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 39039 } }
I20240507 02:11:24.786782 21604 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61eae55814274a1db8dc6dd0fb52a942 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61eae55814274a1db8dc6dd0fb52a942. Latest consensus state: current_term: 1 leader_uuid: "61eae55814274a1db8dc6dd0fb52a942" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61eae55814274a1db8dc6dd0fb52a942" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 39039 } } }
I20240507 02:11:24.786790 21605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61eae55814274a1db8dc6dd0fb52a942 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "61eae55814274a1db8dc6dd0fb52a942" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61eae55814274a1db8dc6dd0fb52a942" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 39039 } } }
I20240507 02:11:24.786857 21604 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61eae55814274a1db8dc6dd0fb52a942 [sys.catalog]: This master's current role is: LEADER
I20240507 02:11:24.786863 21605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61eae55814274a1db8dc6dd0fb52a942 [sys.catalog]: This master's current role is: LEADER
I20240507 02:11:24.787061 21611 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 02:11:24.787503 21611 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240507 02:11:24.791603 21611 catalog_manager.cc:1347] Generated new cluster ID: 1552a872294746ed9017f8ca74ae280f
I20240507 02:11:24.791682 21611 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240507 02:11:24.793699 21602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:24.791131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41255 (local address 127.31.131.126:39039)
0507 02:11:24.791172 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:24.791175 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:24.791337 (+ 162us) server_negotiation.cc:407] Connection header received
0507 02:11:24.791389 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:24.791390 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:24.791401 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:24.791464 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:24.791863 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:24.792300 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:24.792846 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:24.792996 (+ 150us) server_negotiation.cc:662] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:24.793450 (+ 454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:24.793453 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:24.793453 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0507 02:11:24.793459 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:24.793472 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0507 02:11:24.793476 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:24.793534 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:24.793610 (+ 76us) server_negotiation.cc:299] Negotiation successful
0507 02:11:24.793620 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20240507 02:11:24.822670 21611 catalog_manager.cc:1370] Generated new certificate authority record
I20240507 02:11:24.823204 21611 catalog_manager.cc:1504] Loading token signing keys...
I20240507 02:11:24.827286 21611 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 61eae55814274a1db8dc6dd0fb52a942: Generated new TSK 0
I20240507 02:11:24.827488 21611 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240507 02:11:24.840929 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 21531
W20240507 02:11:24.845081 21516 connection.cc:535] client connection to 127.31.131.126:39039 recv error: Network error: failed to read from TLS socket (remote: 127.31.131.126:39039): Connection reset by peer (error 104)
2024-05-07T02:11:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1165 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1715047604443657-32269-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 07 02:11:24 dist-test-slave-v0bg krb5kdc[21630](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 07 02:11:24 dist-test-slave-v0bg krb5kdc[21630](info): set up 2 sockets
May 07 02:11:24 dist-test-slave-v0bg krb5kdc[21630](info): commencing operation
krb5kdc: starting...
W20240507 02:11:25.723443 32269 mini_kdc.cc:121] Time spent starting KDC: real 0.865s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1715047604443657-32269-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1715047604443657-32269-0/krb5kdc/test-user.keytab.
May 07 02:11:25 dist-test-slave-v0bg krb5kdc[21630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047885, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-07T02:11:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-07T02:11:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.131.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab.
Entry for principal kudu/127.31.131.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab.
I20240507 02:11:25.826817 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1715047604443657-32269-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1715047604443657-32269-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1715047604443657-32269-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:40251
--webserver_interface=127.31.131.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab
--principal=kudu/127.31.131.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:44035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.131.126:40251
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:11:25.903986 21643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:11:25.904187 21643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:11:25.904220 21643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:11:25.905493 21643 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240507 02:11:25.905541 21643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:11:25.905562 21643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:11:25.905579 21643 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240507 02:11:25.905598 21643 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240507 02:11:25.907253 21643 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:44035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1715047604443657-32269-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1715047604443657-32269-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.131.126:40251
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab
--principal=kudu/127.31.131.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:40251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.131.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1715047604443657-32269-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:11:25.907491 21643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:11:25.907673 21643 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:11:25.911342 21648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:11:25.911362 21651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:11:25.911523 21649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:25.911475 21643 server_base.cc:1034] running on GCE node
I20240507 02:11:25.911749 21643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:11:25.911958 21643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:11:25.913080 21643 hybrid_clock.cc:648] HybridClock initialized: now 1715047885913068 us; error 25 us; skew 500 ppm
May 07 02:11:25 dist-test-slave-v0bg krb5kdc[21630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047885, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:11:25.915243 21643 init.cc:377] Logged in from keytab as kudu/127.31.131.126@KRBTEST.COM (short username kudu)
I20240507 02:11:25.915611 21643 webserver.cc:460] Webserver started at http://127.31.131.126:36103/ using document root <none> and password file <none>
I20240507 02:11:25.915766 21643 fs_manager.cc:359] Metadata directory not provided
I20240507 02:11:25.915805 21643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:11:25.915904 21643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:11:25.916826 21643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1715047604443657-32269-0/minicluster-data/master-0/data/instance:
uuid: "db460d6249fe48e392601f67f5bfa906"
format_stamp: "Formatted at 2024-05-07 02:11:25 on dist-test-slave-v0bg"
server_key: "300d1735e25798df7000ea2a8bbefad8"
server_key_iv: "bdc5a903ad2f9c621a44e918209f114c"
server_key_version: "encryptionkey@0"
I20240507 02:11:25.917147 21643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1715047604443657-32269-0/minicluster-data/master-0/wal/instance:
uuid: "db460d6249fe48e392601f67f5bfa906"
format_stamp: "Formatted at 2024-05-07 02:11:25 on dist-test-slave-v0bg"
server_key: "300d1735e25798df7000ea2a8bbefad8"
server_key_iv: "bdc5a903ad2f9c621a44e918209f114c"
server_key_version: "encryptionkey@0"
I20240507 02:11:25.918656 21643 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240507 02:11:25.919443 21658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:25.919642 21643 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240507 02:11:25.919728 21643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1715047604443657-32269-0/minicluster-data/master-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1715047604443657-32269-0/minicluster-data/master-0/wal
uuid: "db460d6249fe48e392601f67f5bfa906"
format_stamp: "Formatted at 2024-05-07 02:11:25 on dist-test-slave-v0bg"
server_key: "300d1735e25798df7000ea2a8bbefad8"
server_key_iv: "bdc5a903ad2f9c621a44e918209f114c"
server_key_version: "encryptionkey@0"
I20240507 02:11:25.919796 21643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1715047604443657-32269-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1715047604443657-32269-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1715047604443657-32269-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:11:25.924619 21643 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:11:25.926291 21643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:11:25.926381 21643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:11:25.929430 21643 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.126:40251
I20240507 02:11:25.929486 21710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.126:40251 every 8 connection(s)
I20240507 02:11:25.929723 21643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
I20240507 02:11:25.930477 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 21643
I20240507 02:11:25.930543 21711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:11:25.930619 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1715047604443657-32269-0/minicluster-data/master-0/wal/instance
I20240507 02:11:25.930812 32269 external_mini_cluster.cc:1428] Setting key 1a273d1fc87db2f55a2ac000a194d0f2
I20240507 02:11:25.933087 21714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:25.931299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33429 (local address 127.31.131.126:40251)
0507 02:11:25.931437 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:25.931446 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:25.931463 (+ 17us) server_negotiation.cc:407] Connection header received
0507 02:11:25.931935 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:25.931941 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:25.931975 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:25.932088 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:25.932744 (+ 656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:25.932747 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:25.932747 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0507 02:11:25.932750 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:25.932770 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0507 02:11:25.932775 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:25.932841 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:25.932914 (+ 73us) server_negotiation.cc:299] Negotiation successful
0507 02:11:25.932941 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":58,"threads_started":1}
I20240507 02:11:25.933553 21711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db460d6249fe48e392601f67f5bfa906: Bootstrap starting.
I20240507 02:11:25.933967 32269 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240507 02:11:25.934156 21711 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db460d6249fe48e392601f67f5bfa906: Neither blocks nor log segments found. Creating new log.
I20240507 02:11:25.934465 21711 log.cc:824] T 00000000000000000000000000000000 P db460d6249fe48e392601f67f5bfa906: Log is configured to *not* fsync() on all Append() calls
I20240507 02:11:25.935201 21711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db460d6249fe48e392601f67f5bfa906: No bootstrap required, opened a new log
I20240507 02:11:25.936371 21714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:25.935409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48519 (local address 127.31.131.126:40251)
0507 02:11:25.935446 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:25.935448 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:25.935585 (+ 137us) server_negotiation.cc:407] Connection header received
0507 02:11:25.935656 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:25.935656 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:25.935660 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:25.935702 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:25.936202 (+ 500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:25.936203 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:25.936203 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0507 02:11:25.936206 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:25.936218 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0507 02:11:25.936220 (+ 2us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:25.936259 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:25.936312 (+ 53us) server_negotiation.cc:299] Negotiation successful
0507 02:11:25.936322 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20240507 02:11:25.937166 21711 raft_consensus.cc:348] T 00000000000000000000000000000000 P db460d6249fe48e392601f67f5bfa906 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db460d6249fe48e392601f67f5bfa906" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 40251 } }
I20240507 02:11:25.937289 21711 raft_consensus.cc:374] T 00000000000000000000000000000000 P db460d6249fe48e392601f67f5bfa906 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:11:25.937316 21711 raft_consensus.cc:729] T 00000000000000000000000000000000 P db460d6249fe48e392601f67f5bfa906 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db460d6249fe48e392601f67f5bfa906, State: Initialized, Role: FOLLOWER
I20240507 02:11:25.937407 21711 consensus_queue.cc:260] T 00000000000000000000000000000000 P db460d6249fe48e392601f67f5bfa906 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db460d6249fe48e392601f67f5bfa906" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 40251 } }
I20240507 02:11:25.937474 21711 raft_consensus.cc:388] T 00000000000000000000000000000000 P db460d6249fe48e392601f67f5bfa906 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 02:11:25.937503 21711 raft_consensus.cc:482] T 00000000000000000000000000000000 P db460d6249fe48e392601f67f5bfa906 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 02:11:25.937552 21711 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db460d6249fe48e392601f67f5bfa906 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:11:25.938066 21711 raft_consensus.cc:504] T 00000000000000000000000000000000 P db460d6249fe48e392601f67f5bfa906 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db460d6249fe48e392601f67f5bfa906" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 40251 } }
I20240507 02:11:25.938164 21711 leader_election.cc:302] T 00000000000000000000000000000000 P db460d6249fe48e392601f67f5bfa906 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db460d6249fe48e392601f67f5bfa906; no voters:
I20240507 02:11:25.938303 21711 leader_election.cc:288] T 00000000000000000000000000000000 P db460d6249fe48e392601f67f5bfa906 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240507 02:11:25.938349 21721 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db460d6249fe48e392601f67f5bfa906 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:11:25.938488 21721 raft_consensus.cc:686] T 00000000000000000000000000000000 P db460d6249fe48e392601f67f5bfa906 [term 1 LEADER]: Becoming Leader. State: Replica: db460d6249fe48e392601f67f5bfa906, State: Running, Role: LEADER
I20240507 02:11:25.938534 21711 sys_catalog.cc:564] T 00000000000000000000000000000000 P db460d6249fe48e392601f67f5bfa906 [sys.catalog]: configured and running, proceeding with master startup.
I20240507 02:11:25.938601 21721 consensus_queue.cc:237] T 00000000000000000000000000000000 P db460d6249fe48e392601f67f5bfa906 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db460d6249fe48e392601f67f5bfa906" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 40251 } }
I20240507 02:11:25.938917 21723 sys_catalog.cc:455] T 00000000000000000000000000000000 P db460d6249fe48e392601f67f5bfa906 [sys.catalog]: SysCatalogTable state changed. Reason: New leader db460d6249fe48e392601f67f5bfa906. Latest consensus state: current_term: 1 leader_uuid: "db460d6249fe48e392601f67f5bfa906" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db460d6249fe48e392601f67f5bfa906" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 40251 } } }
I20240507 02:11:25.938978 21723 sys_catalog.cc:458] T 00000000000000000000000000000000 P db460d6249fe48e392601f67f5bfa906 [sys.catalog]: This master's current role is: LEADER
I20240507 02:11:25.938953 21722 sys_catalog.cc:455] T 00000000000000000000000000000000 P db460d6249fe48e392601f67f5bfa906 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db460d6249fe48e392601f67f5bfa906" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db460d6249fe48e392601f67f5bfa906" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 40251 } } }
I20240507 02:11:25.939045 21722 sys_catalog.cc:458] T 00000000000000000000000000000000 P db460d6249fe48e392601f67f5bfa906 [sys.catalog]: This master's current role is: LEADER
I20240507 02:11:25.939185 21726 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 02:11:25.939621 21726 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240507 02:11:25.941058 21726 catalog_manager.cc:1347] Generated new cluster ID: bf3f1e679804452b83c3c0d83e7cd8d6
I20240507 02:11:25.941103 21726 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240507 02:11:25.961715 21726 catalog_manager.cc:1370] Generated new certificate authority record
I20240507 02:11:25.961869 21726 catalog_manager.cc:1504] Loading token signing keys...
I20240507 02:11:25.973289 21726 catalog_manager.cc:5857] T 00000000000000000000000000000000 P db460d6249fe48e392601f67f5bfa906: Generated new TSK 0
I20240507 02:11:25.973470 21726 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240507 02:11:25.978142 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 21643
2024-05-07T02:11:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1136 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1715047604443657-32269-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 07 02:11:26 dist-test-slave-v0bg krb5kdc[21742](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 07 02:11:26 dist-test-slave-v0bg krb5kdc[21742](info): set up 2 sockets
May 07 02:11:26 dist-test-slave-v0bg krb5kdc[21742](info): commencing operation
krb5kdc: starting...
W20240507 02:11:26.863255 32269 mini_kdc.cc:121] Time spent starting KDC: real 0.868s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1715047604443657-32269-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1715047604443657-32269-0/krb5kdc/test-user.keytab.
May 07 02:11:26 dist-test-slave-v0bg krb5kdc[21742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-07T02:11:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-07T02:11:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.131.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab.
Entry for principal kudu/127.31.131.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab.
I20240507 02:11:26.964911 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1715047604443657-32269-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1715047604443657-32269-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1715047604443657-32269-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:43579
--webserver_interface=127.31.131.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab
--principal=kudu/127.31.131.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:43247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.131.126:43579
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:11:27.039976 21755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:11:27.040164 21755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:11:27.040186 21755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:11:27.041437 21755 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240507 02:11:27.041476 21755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:11:27.041488 21755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:11:27.041500 21755 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240507 02:11:27.041528 21755 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240507 02:11:27.043102 21755 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:43247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1715047604443657-32269-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1715047604443657-32269-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.131.126:43579
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab
--principal=kudu/127.31.131.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:43579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.131.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1715047604443657-32269-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:11:27.043293 21755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:11:27.043457 21755 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:11:27.047029 21761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:27.047128 21755 server_base.cc:1034] running on GCE node
W20240507 02:11:27.047065 21760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:11:27.047319 21763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:27.047538 21755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:11:27.047732 21755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:11:27.048857 21755 hybrid_clock.cc:648] HybridClock initialized: now 1715047887048844 us; error 29 us; skew 500 ppm
May 07 02:11:27 dist-test-slave-v0bg krb5kdc[21742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047887, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:11:27.051128 21755 init.cc:377] Logged in from keytab as kudu/127.31.131.126@KRBTEST.COM (short username kudu)
I20240507 02:11:27.051496 21755 webserver.cc:460] Webserver started at http://127.31.131.126:35205/ using document root <none> and password file <none>
I20240507 02:11:27.051662 21755 fs_manager.cc:359] Metadata directory not provided
I20240507 02:11:27.051698 21755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:11:27.051795 21755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:11:27.052702 21755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1715047604443657-32269-0/minicluster-data/master-0/data/instance:
uuid: "914bcdb41560417c8fd9e875c6c8ea60"
format_stamp: "Formatted at 2024-05-07 02:11:27 on dist-test-slave-v0bg"
server_key: "9e17185c5fb74e47d1fdddd68c74494f"
server_key_iv: "205336ed3bf4c5a6cc48d12dae5816e2"
server_key_version: "encryptionkey@0"
I20240507 02:11:27.052965 21755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1715047604443657-32269-0/minicluster-data/master-0/wal/instance:
uuid: "914bcdb41560417c8fd9e875c6c8ea60"
format_stamp: "Formatted at 2024-05-07 02:11:27 on dist-test-slave-v0bg"
server_key: "9e17185c5fb74e47d1fdddd68c74494f"
server_key_iv: "205336ed3bf4c5a6cc48d12dae5816e2"
server_key_version: "encryptionkey@0"
I20240507 02:11:27.054003 21755 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240507 02:11:27.054615 21770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:27.054761 21755 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240507 02:11:27.054826 21755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1715047604443657-32269-0/minicluster-data/master-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1715047604443657-32269-0/minicluster-data/master-0/wal
uuid: "914bcdb41560417c8fd9e875c6c8ea60"
format_stamp: "Formatted at 2024-05-07 02:11:27 on dist-test-slave-v0bg"
server_key: "9e17185c5fb74e47d1fdddd68c74494f"
server_key_iv: "205336ed3bf4c5a6cc48d12dae5816e2"
server_key_version: "encryptionkey@0"
I20240507 02:11:27.054885 21755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1715047604443657-32269-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1715047604443657-32269-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1715047604443657-32269-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:11:27.070175 21755 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:11:27.072631 21755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:11:27.072741 21755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:11:27.076602 21755 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.126:43579
I20240507 02:11:27.076639 21822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.126:43579 every 8 connection(s)
I20240507 02:11:27.076931 21755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
I20240507 02:11:27.077600 21823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:11:27.078492 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 21755
I20240507 02:11:27.078608 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1715047604443657-32269-0/minicluster-data/master-0/wal/instance
I20240507 02:11:27.078768 32269 external_mini_cluster.cc:1428] Setting key b43d3276759d646dfbd7f7fca65e6365
I20240507 02:11:27.080036 21823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 914bcdb41560417c8fd9e875c6c8ea60: Bootstrap starting.
I20240507 02:11:27.080572 21823 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 914bcdb41560417c8fd9e875c6c8ea60: Neither blocks nor log segments found. Creating new log.
I20240507 02:11:27.080806 21823 log.cc:824] T 00000000000000000000000000000000 P 914bcdb41560417c8fd9e875c6c8ea60: Log is configured to *not* fsync() on all Append() calls
I20240507 02:11:27.081454 21823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 914bcdb41560417c8fd9e875c6c8ea60: No bootstrap required, opened a new log
I20240507 02:11:27.083187 21823 raft_consensus.cc:348] T 00000000000000000000000000000000 P 914bcdb41560417c8fd9e875c6c8ea60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "914bcdb41560417c8fd9e875c6c8ea60" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 43579 } }
I20240507 02:11:27.083302 21823 raft_consensus.cc:374] T 00000000000000000000000000000000 P 914bcdb41560417c8fd9e875c6c8ea60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:11:27.083318 21823 raft_consensus.cc:729] T 00000000000000000000000000000000 P 914bcdb41560417c8fd9e875c6c8ea60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 914bcdb41560417c8fd9e875c6c8ea60, State: Initialized, Role: FOLLOWER
I20240507 02:11:27.083401 21823 consensus_queue.cc:260] T 00000000000000000000000000000000 P 914bcdb41560417c8fd9e875c6c8ea60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "914bcdb41560417c8fd9e875c6c8ea60" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 43579 } }
I20240507 02:11:27.083457 21823 raft_consensus.cc:388] T 00000000000000000000000000000000 P 914bcdb41560417c8fd9e875c6c8ea60 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 02:11:27.083478 21823 raft_consensus.cc:482] T 00000000000000000000000000000000 P 914bcdb41560417c8fd9e875c6c8ea60 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 02:11:27.083521 21823 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 914bcdb41560417c8fd9e875c6c8ea60 [term 0 FOLLOWER]: Advancing to term 1
May 07 02:11:27 dist-test-slave-v0bg krb5kdc[21742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:11:27.083966 21823 raft_consensus.cc:504] T 00000000000000000000000000000000 P 914bcdb41560417c8fd9e875c6c8ea60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "914bcdb41560417c8fd9e875c6c8ea60" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 43579 } }
I20240507 02:11:27.084040 21823 leader_election.cc:302] T 00000000000000000000000000000000 P 914bcdb41560417c8fd9e875c6c8ea60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 914bcdb41560417c8fd9e875c6c8ea60; no voters:
I20240507 02:11:27.084179 21823 leader_election.cc:288] T 00000000000000000000000000000000 P 914bcdb41560417c8fd9e875c6c8ea60 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240507 02:11:27.084271 21828 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 914bcdb41560417c8fd9e875c6c8ea60 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:11:27.084403 21823 sys_catalog.cc:564] T 00000000000000000000000000000000 P 914bcdb41560417c8fd9e875c6c8ea60 [sys.catalog]: configured and running, proceeding with master startup.
I20240507 02:11:27.084475 21828 raft_consensus.cc:686] T 00000000000000000000000000000000 P 914bcdb41560417c8fd9e875c6c8ea60 [term 1 LEADER]: Becoming Leader. State: Replica: 914bcdb41560417c8fd9e875c6c8ea60, State: Running, Role: LEADER
I20240507 02:11:27.084604 21828 consensus_queue.cc:237] T 00000000000000000000000000000000 P 914bcdb41560417c8fd9e875c6c8ea60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "914bcdb41560417c8fd9e875c6c8ea60" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 43579 } }
I20240507 02:11:27.084941 21829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 914bcdb41560417c8fd9e875c6c8ea60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "914bcdb41560417c8fd9e875c6c8ea60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "914bcdb41560417c8fd9e875c6c8ea60" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 43579 } } }
I20240507 02:11:27.085033 21829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 914bcdb41560417c8fd9e875c6c8ea60 [sys.catalog]: This master's current role is: LEADER
I20240507 02:11:27.085206 21837 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 02:11:27.085206 21830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 914bcdb41560417c8fd9e875c6c8ea60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 914bcdb41560417c8fd9e875c6c8ea60. Latest consensus state: current_term: 1 leader_uuid: "914bcdb41560417c8fd9e875c6c8ea60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "914bcdb41560417c8fd9e875c6c8ea60" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 43579 } } }
I20240507 02:11:27.085341 21830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 914bcdb41560417c8fd9e875c6c8ea60 [sys.catalog]: This master's current role is: LEADER
I20240507 02:11:27.085911 21837 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240507 02:11:27.087253 21837 catalog_manager.cc:1347] Generated new cluster ID: 18d1927a82d54a44b234fad5c6488019
I20240507 02:11:27.087301 21837 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240507 02:11:27.088425 21826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:27.079203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42227 (local address 127.31.131.126:43579)
0507 02:11:27.079379 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:27.079391 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:27.079414 (+ 23us) server_negotiation.cc:407] Connection header received
0507 02:11:27.079906 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:27.079914 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:27.079968 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:27.080125 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:27.080778 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:27.081808 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:27.082348 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:27.082597 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:27.084454 (+ 1857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:27.084462 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:27.084462 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:11:27.084481 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:27.087252 (+ 2771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:27.087555 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:27.087558 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:27.087558 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:27.087601 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:27.087736 (+ 135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:27.087737 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:27.087737 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:27.087913 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:27.088044 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:27.088147 (+ 103us) server_negotiation.cc:299] Negotiation successful
0507 02:11:27.088261 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":91,"threads_started":1}
I20240507 02:11:27.089198 32269 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240507 02:11:27.099592 21826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:27.094665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48017 (local address 127.31.131.126:43579)
0507 02:11:27.094702 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:27.094703 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:27.094850 (+ 147us) server_negotiation.cc:407] Connection header received
0507 02:11:27.094905 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:27.094906 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:27.094915 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:27.094957 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:27.095558 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:27.095983 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:27.096458 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:27.096644 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:27.097502 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:27.097506 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:27.097506 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:11:27.097512 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:27.098736 (+ 1224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:27.098996 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:27.098997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:27.098997 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:27.099018 (+ 21us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:27.099156 (+ 138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:27.099156 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:27.099157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:27.099239 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:27.099318 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:27.099413 (+ 95us) server_negotiation.cc:299] Negotiation successful
0507 02:11:27.099489 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20240507 02:11:27.105876 21837 catalog_manager.cc:1370] Generated new certificate authority record
I20240507 02:11:27.106412 21837 catalog_manager.cc:1504] Loading token signing keys...
I20240507 02:11:27.113883 21837 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 914bcdb41560417c8fd9e875c6c8ea60: Generated new TSK 0
I20240507 02:11:27.114068 21837 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240507 02:11:27.116304 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 21755
2024-05-07T02:11:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1138 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1715047604443657-32269-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 07 02:11:27 dist-test-slave-v0bg krb5kdc[21854](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 07 02:11:27 dist-test-slave-v0bg krb5kdc[21854](info): set up 2 sockets
May 07 02:11:27 dist-test-slave-v0bg krb5kdc[21854](info): commencing operation
krb5kdc: starting...
W20240507 02:11:27.999347 32269 mini_kdc.cc:121] Time spent starting KDC: real 0.866s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1715047604443657-32269-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1715047604443657-32269-0/krb5kdc/test-user.keytab.
May 07 02:11:28 dist-test-slave-v0bg krb5kdc[21854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-07T02:11:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-07T02:11:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.131.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab.
Entry for principal kudu/127.31.131.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab.
I20240507 02:11:28.102802 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1715047604443657-32269-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1715047604443657-32269-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1715047604443657-32269-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:44033
--webserver_interface=127.31.131.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab
--principal=kudu/127.31.131.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:43349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.131.126:44033
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:11:28.178642 21867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:11:28.178833 21867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:11:28.178865 21867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:11:28.180150 21867 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240507 02:11:28.180193 21867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:11:28.180209 21867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:11:28.180224 21867 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240507 02:11:28.180238 21867 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240507 02:11:28.181802 21867 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:43349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1715047604443657-32269-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1715047604443657-32269-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.131.126:44033
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab
--principal=kudu/127.31.131.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:44033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.131.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1715047604443657-32269-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:11:28.182035 21867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:11:28.182207 21867 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:11:28.185858 21873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:11:28.185909 21872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:28.186007 21867 server_base.cc:1034] running on GCE node
W20240507 02:11:28.186149 21875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:28.186357 21867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:11:28.186635 21867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:11:28.187790 21867 hybrid_clock.cc:648] HybridClock initialized: now 1715047888187770 us; error 37 us; skew 500 ppm
May 07 02:11:28 dist-test-slave-v0bg krb5kdc[21854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047888, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:11:28.194185 21867 init.cc:377] Logged in from keytab as kudu/127.31.131.126@KRBTEST.COM (short username kudu)
I20240507 02:11:28.194679 21867 webserver.cc:460] Webserver started at http://127.31.131.126:43539/ using document root <none> and password file <none>
I20240507 02:11:28.194870 21867 fs_manager.cc:359] Metadata directory not provided
I20240507 02:11:28.194912 21867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:11:28.195022 21867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:11:28.195973 21867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1715047604443657-32269-0/minicluster-data/master-0/data/instance:
uuid: "53fc3eb40d994513b205e59cc89bea1b"
format_stamp: "Formatted at 2024-05-07 02:11:28 on dist-test-slave-v0bg"
server_key: "7e64c0c11ee3f850166aa1383c1b0c14"
server_key_iv: "3af591c7dd3ccf1c6d6930a292421986"
server_key_version: "encryptionkey@0"
I20240507 02:11:28.196264 21867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1715047604443657-32269-0/minicluster-data/master-0/wal/instance:
uuid: "53fc3eb40d994513b205e59cc89bea1b"
format_stamp: "Formatted at 2024-05-07 02:11:28 on dist-test-slave-v0bg"
server_key: "7e64c0c11ee3f850166aa1383c1b0c14"
server_key_iv: "3af591c7dd3ccf1c6d6930a292421986"
server_key_version: "encryptionkey@0"
I20240507 02:11:28.197360 21867 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20240507 02:11:28.197957 21882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:28.198103 21867 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240507 02:11:28.198177 21867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1715047604443657-32269-0/minicluster-data/master-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1715047604443657-32269-0/minicluster-data/master-0/wal
uuid: "53fc3eb40d994513b205e59cc89bea1b"
format_stamp: "Formatted at 2024-05-07 02:11:28 on dist-test-slave-v0bg"
server_key: "7e64c0c11ee3f850166aa1383c1b0c14"
server_key_iv: "3af591c7dd3ccf1c6d6930a292421986"
server_key_version: "encryptionkey@0"
I20240507 02:11:28.198233 21867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1715047604443657-32269-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1715047604443657-32269-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1715047604443657-32269-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:11:28.210669 21867 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:11:28.212620 21867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:11:28.212733 21867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:11:28.216128 21867 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.126:44033
I20240507 02:11:28.216161 21934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.126:44033 every 8 connection(s)
I20240507 02:11:28.216454 21867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
I20240507 02:11:28.217156 21935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:11:28.219197 21935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53fc3eb40d994513b205e59cc89bea1b: Bootstrap starting.
I20240507 02:11:28.219682 21935 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53fc3eb40d994513b205e59cc89bea1b: Neither blocks nor log segments found. Creating new log.
I20240507 02:11:28.219884 21935 log.cc:824] T 00000000000000000000000000000000 P 53fc3eb40d994513b205e59cc89bea1b: Log is configured to *not* fsync() on all Append() calls
I20240507 02:11:28.220440 21935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53fc3eb40d994513b205e59cc89bea1b: No bootstrap required, opened a new log
I20240507 02:11:28.221753 21935 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53fc3eb40d994513b205e59cc89bea1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53fc3eb40d994513b205e59cc89bea1b" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 44033 } }
I20240507 02:11:28.221869 21935 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53fc3eb40d994513b205e59cc89bea1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:11:28.221885 21935 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53fc3eb40d994513b205e59cc89bea1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53fc3eb40d994513b205e59cc89bea1b, State: Initialized, Role: FOLLOWER
I20240507 02:11:28.221978 21935 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53fc3eb40d994513b205e59cc89bea1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53fc3eb40d994513b205e59cc89bea1b" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 44033 } }
I20240507 02:11:28.222031 21935 raft_consensus.cc:388] T 00000000000000000000000000000000 P 53fc3eb40d994513b205e59cc89bea1b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 02:11:28.222050 21935 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53fc3eb40d994513b205e59cc89bea1b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 02:11:28.222075 21935 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53fc3eb40d994513b205e59cc89bea1b [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:11:28.222523 21935 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53fc3eb40d994513b205e59cc89bea1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53fc3eb40d994513b205e59cc89bea1b" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 44033 } }
I20240507 02:11:28.222627 21935 leader_election.cc:302] T 00000000000000000000000000000000 P 53fc3eb40d994513b205e59cc89bea1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53fc3eb40d994513b205e59cc89bea1b; no voters:
I20240507 02:11:28.222738 21935 leader_election.cc:288] T 00000000000000000000000000000000 P 53fc3eb40d994513b205e59cc89bea1b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240507 02:11:28.222788 21938 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53fc3eb40d994513b205e59cc89bea1b [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:11:28.222944 21938 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53fc3eb40d994513b205e59cc89bea1b [term 1 LEADER]: Becoming Leader. State: Replica: 53fc3eb40d994513b205e59cc89bea1b, State: Running, Role: LEADER
I20240507 02:11:28.223031 21938 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53fc3eb40d994513b205e59cc89bea1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53fc3eb40d994513b205e59cc89bea1b" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 44033 } }
I20240507 02:11:28.223289 21935 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53fc3eb40d994513b205e59cc89bea1b [sys.catalog]: configured and running, proceeding with master startup.
I20240507 02:11:28.223358 21939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53fc3eb40d994513b205e59cc89bea1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "53fc3eb40d994513b205e59cc89bea1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53fc3eb40d994513b205e59cc89bea1b" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 44033 } } }
I20240507 02:11:28.223430 21939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53fc3eb40d994513b205e59cc89bea1b [sys.catalog]: This master's current role is: LEADER
I20240507 02:11:28.223575 21940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53fc3eb40d994513b205e59cc89bea1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53fc3eb40d994513b205e59cc89bea1b. Latest consensus state: current_term: 1 leader_uuid: "53fc3eb40d994513b205e59cc89bea1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53fc3eb40d994513b205e59cc89bea1b" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 44033 } } }
I20240507 02:11:28.223616 21942 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 02:11:28.223693 21940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53fc3eb40d994513b205e59cc89bea1b [sys.catalog]: This master's current role is: LEADER
I20240507 02:11:28.224166 21942 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240507 02:11:28.225579 21942 catalog_manager.cc:1347] Generated new cluster ID: 361b1ad22ac247b49950b5dfe431eec1
I20240507 02:11:28.225626 21942 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240507 02:11:28.226670 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 21867
I20240507 02:11:28.226763 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1715047604443657-32269-0/minicluster-data/master-0/wal/instance
I20240507 02:11:28.226917 32269 external_mini_cluster.cc:1428] Setting key 544eeaeb34c9d27a3c408b121631263e
I20240507 02:11:28.231654 21958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:28.227354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59623 (local address 127.31.131.126:44033)
0507 02:11:28.227520 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:28.227531 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:28.227622 (+ 91us) server_negotiation.cc:407] Connection header received
0507 02:11:28.227749 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:28.227755 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:28.227815 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:28.227949 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:28.228314 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:28.229478 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:28.230187 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:28.230423 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:28.230768 (+ 345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:28.230775 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:28.230775 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0507 02:11:28.231269 (+ 494us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:28.231286 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0507 02:11:28.231295 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:28.231371 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:28.231435 (+ 64us) server_negotiation.cc:299] Negotiation successful
0507 02:11:28.231478 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":78,"threads_started":1}
I20240507 02:11:28.232638 32269 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240507 02:11:28.236599 21958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:28.233945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42121 (local address 127.31.131.126:44033)
0507 02:11:28.233980 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:28.233982 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:28.234098 (+ 116us) server_negotiation.cc:407] Connection header received
0507 02:11:28.234127 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:28.234128 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:28.234138 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:28.234212 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:28.234501 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:28.235215 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:28.235865 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:28.236050 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:28.236368 (+ 318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:28.236371 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:28.236371 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0507 02:11:28.236375 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:28.236387 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0507 02:11:28.236390 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:28.236447 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:28.236501 (+ 54us) server_negotiation.cc:299] Negotiation successful
0507 02:11:28.236528 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20240507 02:11:28.248483 21942 catalog_manager.cc:1370] Generated new certificate authority record
I20240507 02:11:28.249254 21942 catalog_manager.cc:1504] Loading token signing keys...
I20240507 02:11:28.263651 21942 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 53fc3eb40d994513b205e59cc89bea1b: Generated new TSK 0
I20240507 02:11:28.264513 21942 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240507 02:11:28.269990 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 21867
2024-05-07T02:11:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1154 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1715047604443657-32269-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 07 02:11:28 dist-test-slave-v0bg krb5kdc[21966](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 07 02:11:28 dist-test-slave-v0bg krb5kdc[21966](info): set up 2 sockets
May 07 02:11:28 dist-test-slave-v0bg krb5kdc[21966](info): commencing operation
krb5kdc: starting...
W20240507 02:11:29.163401 32269 mini_kdc.cc:121] Time spent starting KDC: real 0.876s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1715047604443657-32269-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1715047604443657-32269-0/krb5kdc/test-user.keytab.
May 07 02:11:29 dist-test-slave-v0bg krb5kdc[21966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-07T02:11:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-07T02:11:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.131.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab.
Entry for principal kudu/127.31.131.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab.
I20240507 02:11:29.270856 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1715047604443657-32269-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1715047604443657-32269-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1715047604443657-32269-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:39483
--webserver_interface=127.31.131.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab
--principal=kudu/127.31.131.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:34867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.131.126:39483
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:11:29.346385 21979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:11:29.346580 21979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:11:29.346606 21979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:11:29.347860 21979 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240507 02:11:29.347900 21979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:11:29.347913 21979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:11:29.347923 21979 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240507 02:11:29.347936 21979 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240507 02:11:29.349503 21979 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:34867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1715047604443657-32269-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1715047604443657-32269-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.131.126:39483
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab
--principal=kudu/127.31.131.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:39483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.131.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1715047604443657-32269-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:11:29.349681 21979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:11:29.349850 21979 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:11:29.353528 21984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:11:29.353562 21985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:29.353600 21979 server_base.cc:1034] running on GCE node
W20240507 02:11:29.353716 21987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:29.353907 21979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:11:29.354144 21979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:11:29.355291 21979 hybrid_clock.cc:648] HybridClock initialized: now 1715047889355274 us; error 35 us; skew 500 ppm
May 07 02:11:29 dist-test-slave-v0bg krb5kdc[21966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047889, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:11:29.357460 21979 init.cc:377] Logged in from keytab as kudu/127.31.131.126@KRBTEST.COM (short username kudu)
I20240507 02:11:29.357808 21979 webserver.cc:460] Webserver started at http://127.31.131.126:43407/ using document root <none> and password file <none>
I20240507 02:11:29.357945 21979 fs_manager.cc:359] Metadata directory not provided
I20240507 02:11:29.357977 21979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:11:29.358052 21979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:11:29.358983 21979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1715047604443657-32269-0/minicluster-data/master-0/data/instance:
uuid: "32d9f25d6a40464cac527ee3133bc6ab"
format_stamp: "Formatted at 2024-05-07 02:11:29 on dist-test-slave-v0bg"
server_key: "c29bbf3d66983511089f04730f9c24c6"
server_key_iv: "f9f1ac341d186284898a828b1f859056"
server_key_version: "encryptionkey@0"
I20240507 02:11:29.359234 21979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1715047604443657-32269-0/minicluster-data/master-0/wal/instance:
uuid: "32d9f25d6a40464cac527ee3133bc6ab"
format_stamp: "Formatted at 2024-05-07 02:11:29 on dist-test-slave-v0bg"
server_key: "c29bbf3d66983511089f04730f9c24c6"
server_key_iv: "f9f1ac341d186284898a828b1f859056"
server_key_version: "encryptionkey@0"
I20240507 02:11:29.360230 21979 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240507 02:11:29.360786 21994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:29.360927 21979 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240507 02:11:29.360992 21979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1715047604443657-32269-0/minicluster-data/master-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1715047604443657-32269-0/minicluster-data/master-0/wal
uuid: "32d9f25d6a40464cac527ee3133bc6ab"
format_stamp: "Formatted at 2024-05-07 02:11:29 on dist-test-slave-v0bg"
server_key: "c29bbf3d66983511089f04730f9c24c6"
server_key_iv: "f9f1ac341d186284898a828b1f859056"
server_key_version: "encryptionkey@0"
I20240507 02:11:29.361052 21979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1715047604443657-32269-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1715047604443657-32269-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1715047604443657-32269-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:11:29.364419 21979 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:11:29.365897 21979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:11:29.365991 21979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:11:29.369087 21979 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.126:39483
I20240507 02:11:29.369134 22046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.126:39483 every 8 connection(s)
I20240507 02:11:29.369364 21979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
I20240507 02:11:29.370100 22047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:11:29.372057 22047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32d9f25d6a40464cac527ee3133bc6ab: Bootstrap starting.
I20240507 02:11:29.372555 22047 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32d9f25d6a40464cac527ee3133bc6ab: Neither blocks nor log segments found. Creating new log.
I20240507 02:11:29.372771 22047 log.cc:824] T 00000000000000000000000000000000 P 32d9f25d6a40464cac527ee3133bc6ab: Log is configured to *not* fsync() on all Append() calls
I20240507 02:11:29.373322 22047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32d9f25d6a40464cac527ee3133bc6ab: No bootstrap required, opened a new log
I20240507 02:11:29.374459 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 21979
I20240507 02:11:29.374616 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1715047604443657-32269-0/minicluster-data/master-0/wal/instance
I20240507 02:11:29.374770 32269 external_mini_cluster.cc:1428] Setting key e8b195174cb21f3b22b52e5925b60eec
I20240507 02:11:29.374876 22047 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32d9f25d6a40464cac527ee3133bc6ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32d9f25d6a40464cac527ee3133bc6ab" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 39483 } }
I20240507 02:11:29.375007 22047 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32d9f25d6a40464cac527ee3133bc6ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 02:11:29.375034 22047 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32d9f25d6a40464cac527ee3133bc6ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32d9f25d6a40464cac527ee3133bc6ab, State: Initialized, Role: FOLLOWER
I20240507 02:11:29.375159 22047 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32d9f25d6a40464cac527ee3133bc6ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32d9f25d6a40464cac527ee3133bc6ab" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 39483 } }
I20240507 02:11:29.375237 22047 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32d9f25d6a40464cac527ee3133bc6ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 02:11:29.375298 22047 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32d9f25d6a40464cac527ee3133bc6ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 02:11:29.375350 22047 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32d9f25d6a40464cac527ee3133bc6ab [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:11:29.375990 22047 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32d9f25d6a40464cac527ee3133bc6ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32d9f25d6a40464cac527ee3133bc6ab" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 39483 } }
I20240507 02:11:29.376101 22047 leader_election.cc:302] T 00000000000000000000000000000000 P 32d9f25d6a40464cac527ee3133bc6ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32d9f25d6a40464cac527ee3133bc6ab; no voters:
I20240507 02:11:29.376250 22047 leader_election.cc:288] T 00000000000000000000000000000000 P 32d9f25d6a40464cac527ee3133bc6ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20240507 02:11:29.376406 22052 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32d9f25d6a40464cac527ee3133bc6ab [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:11:29.376502 22047 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32d9f25d6a40464cac527ee3133bc6ab [sys.catalog]: configured and running, proceeding with master startup.
I20240507 02:11:29.376555 22052 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32d9f25d6a40464cac527ee3133bc6ab [term 1 LEADER]: Becoming Leader. State: Replica: 32d9f25d6a40464cac527ee3133bc6ab, State: Running, Role: LEADER
I20240507 02:11:29.376652 22052 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32d9f25d6a40464cac527ee3133bc6ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32d9f25d6a40464cac527ee3133bc6ab" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 39483 } }
I20240507 02:11:29.376821 22051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:29.375168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51851 (local address 127.31.131.126:39483)
0507 02:11:29.375341 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:29.375353 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:29.375377 (+ 24us) server_negotiation.cc:407] Connection header received
0507 02:11:29.375480 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:29.375484 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:29.375506 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:29.375617 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:29.376266 (+ 649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:29.376269 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:29.376269 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0507 02:11:29.376273 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:29.376296 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0507 02:11:29.376304 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:29.376383 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:29.376638 (+ 255us) server_negotiation.cc:299] Negotiation successful
0507 02:11:29.376673 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":77,"threads_started":1}
I20240507 02:11:29.376925 22052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32d9f25d6a40464cac527ee3133bc6ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32d9f25d6a40464cac527ee3133bc6ab. Latest consensus state: current_term: 1 leader_uuid: "32d9f25d6a40464cac527ee3133bc6ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32d9f25d6a40464cac527ee3133bc6ab" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 39483 } } }
I20240507 02:11:29.377007 22052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32d9f25d6a40464cac527ee3133bc6ab [sys.catalog]: This master's current role is: LEADER
I20240507 02:11:29.377115 22053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32d9f25d6a40464cac527ee3133bc6ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32d9f25d6a40464cac527ee3133bc6ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32d9f25d6a40464cac527ee3133bc6ab" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 39483 } } }
I20240507 02:11:29.377243 22053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32d9f25d6a40464cac527ee3133bc6ab [sys.catalog]: This master's current role is: LEADER
I20240507 02:11:29.377624 32269 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20240507 02:11:29.378312 22067 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 32d9f25d6a40464cac527ee3133bc6ab: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240507 02:11:29.378816 22067 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240507 02:11:29.378895 22058 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 02:11:29.379072 22058 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240507 02:11:29.380241 22051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:29.379057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34461 (local address 127.31.131.126:39483)
0507 02:11:29.379092 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:29.379094 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:29.379270 (+ 176us) server_negotiation.cc:407] Connection header received
0507 02:11:29.379317 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:29.379317 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:29.379323 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:29.379396 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:29.380012 (+ 616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:29.380013 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:29.380013 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0507 02:11:29.380017 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:29.380030 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0507 02:11:29.380034 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:29.380093 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:29.380163 (+ 70us) server_negotiation.cc:299] Negotiation successful
0507 02:11:29.380178 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20240507 02:11:29.380542 22058 catalog_manager.cc:1347] Generated new cluster ID: 17812b6d4b2e442ba6a84433f47c4c1f
I20240507 02:11:29.380592 22058 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240507 02:11:29.391008 22058 catalog_manager.cc:1370] Generated new certificate authority record
I20240507 02:11:29.391156 22058 catalog_manager.cc:1504] Loading token signing keys...
I20240507 02:11:29.394289 22058 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 32d9f25d6a40464cac527ee3133bc6ab: Generated new TSK 0
I20240507 02:11:29.394459 22058 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240507 02:11:29.397351 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 21979
2024-05-07T02:11:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1127 ms)
[----------] 18 tests from AuthTokenIssuingTest (20476 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715047604443657-32269-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 07 02:11:29 dist-test-slave-v0bg krb5kdc[22078](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 07 02:11:29 dist-test-slave-v0bg krb5kdc[22078](info): set up 2 sockets
May 07 02:11:29 dist-test-slave-v0bg krb5kdc[22078](info): commencing operation
krb5kdc: starting...
W20240507 02:11:30.283401 32269 mini_kdc.cc:121] Time spent starting KDC: real 0.868s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715047604443657-32269-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715047604443657-32269-0/krb5kdc/test-user.keytab.
May 07 02:11:30 dist-test-slave-v0bg krb5kdc[22078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-07T02:11:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-07T02:11:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.131.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab.
Entry for principal kudu/127.31.131.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab.
I20240507 02:11:30.385025 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715047604443657-32269-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715047604443657-32269-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715047604443657-32269-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:39745
--webserver_interface=127.31.131.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab
--principal=kudu/127.31.131.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:33573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.131.126:39745,127.31.131.125:45921,127.31.131.124:42983
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:11:30.460598 22091 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20240507 02:11:30.460803 22091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:11:30.460825 22091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:11:30.460850 22091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:11:30.462131 22091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240507 02:11:30.462173 22091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:11:30.462190 22091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:11:30.462205 22091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240507 02:11:30.462219 22091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240507 02:11:30.463692 22091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:33573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715047604443657-32269-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715047604443657-32269-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.131.126:39745,127.31.131.125:45921,127.31.131.124:42983
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab
--principal=kudu/127.31.131.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:39745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.131.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715047604443657-32269-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:11:30.463899 22091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:11:30.464085 22091 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:11:30.467876 22096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:30.468012 22091 server_base.cc:1034] running on GCE node
W20240507 02:11:30.468102 22099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:11:30.467876 22097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:30.468353 22091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:11:30.468575 22091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:11:30.469695 22091 hybrid_clock.cc:648] HybridClock initialized: now 1715047890469681 us; error 33 us; skew 500 ppm
May 07 02:11:30 dist-test-slave-v0bg krb5kdc[22078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047890, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:11:30.471910 22091 init.cc:377] Logged in from keytab as kudu/127.31.131.126@KRBTEST.COM (short username kudu)
I20240507 02:11:30.472313 22091 webserver.cc:460] Webserver started at http://127.31.131.126:46707/ using document root <none> and password file <none>
I20240507 02:11:30.472478 22091 fs_manager.cc:359] Metadata directory not provided
I20240507 02:11:30.472524 22091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:11:30.472625 22091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:11:30.473570 22091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715047604443657-32269-0/minicluster-data/master-0/data/instance:
uuid: "09beb00ac8bf4b96a5304facaf05b0b7"
format_stamp: "Formatted at 2024-05-07 02:11:30 on dist-test-slave-v0bg"
server_key: "c57210149ad508984dea92343007262a"
server_key_iv: "8171abcd278d14a90ecea79a4e3686d0"
server_key_version: "encryptionkey@0"
I20240507 02:11:30.473858 22091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715047604443657-32269-0/minicluster-data/master-0/wal/instance:
uuid: "09beb00ac8bf4b96a5304facaf05b0b7"
format_stamp: "Formatted at 2024-05-07 02:11:30 on dist-test-slave-v0bg"
server_key: "c57210149ad508984dea92343007262a"
server_key_iv: "8171abcd278d14a90ecea79a4e3686d0"
server_key_version: "encryptionkey@0"
I20240507 02:11:30.474982 22091 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240507 02:11:30.475572 22106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:30.475713 22091 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240507 02:11:30.475782 22091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715047604443657-32269-0/minicluster-data/master-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715047604443657-32269-0/minicluster-data/master-0/wal
uuid: "09beb00ac8bf4b96a5304facaf05b0b7"
format_stamp: "Formatted at 2024-05-07 02:11:30 on dist-test-slave-v0bg"
server_key: "c57210149ad508984dea92343007262a"
server_key_iv: "8171abcd278d14a90ecea79a4e3686d0"
server_key_version: "encryptionkey@0"
I20240507 02:11:30.475836 22091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715047604443657-32269-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715047604443657-32269-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715047604443657-32269-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:11:30.490496 22091 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:11:30.492245 22091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:11:30.492368 22091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:11:30.495983 22091 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.126:39745
I20240507 02:11:30.496013 22158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.126:39745 every 8 connection(s)
I20240507 02:11:30.496326 22091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
I20240507 02:11:30.497114 22159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:11:30.498517 22159 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.131.126" port: 39745 } has no permanent_uuid. Determining permanent_uuid...
I20240507 02:11:30.498759 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 22091
I20240507 02:11:30.498857 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715047604443657-32269-0/minicluster-data/master-0/wal/instance
I20240507 02:11:30.499019 32269 external_mini_cluster.cc:1428] Setting key ef583a3eb0ff22b267c0b81e1a2d0c00
May 07 02:11:30 dist-test-slave-v0bg krb5kdc[22078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047890, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.126@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.131.125@KRBTEST.COM; defaulting to no policy
I20240507 02:11:30.507315 22162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:30.499330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48892 (local address 127.31.131.126:39745)
0507 02:11:30.499488 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:30.499498 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:30.499584 (+ 86us) server_negotiation.cc:407] Connection header received
0507 02:11:30.500209 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:30.500210 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:30.500227 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:30.500311 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:30.501027 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:30.501673 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:30.502371 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:30.502524 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:30.504583 (+ 2059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:30.504591 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:30.504592 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:11:30.504597 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:30.506026 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:30.506326 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:30.506329 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:30.506330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:30.506364 (+ 34us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:30.506565 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:30.506566 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:30.506566 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:30.506739 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:30.506814 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:30.506974 (+ 160us) server_negotiation.cc:299] Negotiation successful
0507 02:11:30.507082 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":61,"threads_started":1}
I20240507 02:11:30.507344 22161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:30.499288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:39745 (local address 127.0.0.1:48892)
0507 02:11:30.499453 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:30.499489 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0507 02:11:30.500074 (+ 585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:11:30.500327 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:11:30.500330 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:11:30.500673 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:11:30.500922 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:30.500928 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:30.501756 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:11:30.501757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:11:30.502298 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:30.502300 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:30.502453 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:30.503001 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:11:30.503024 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:11:30.504503 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:11:30.506098 (+ 1595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:11:30.506103 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:11:30.506106 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:11:30.506261 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:11:30.506416 (+ 155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:11:30.506417 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:11:30.506417 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:11:30.506498 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:11:30.506797 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:11:30.506800 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:11:30.506859 (+ 59us) client_negotiation.cc:769] Sending connection context
0507 02:11:30.506945 (+ 86us) client_negotiation.cc:240] Negotiation successful
0507 02:11:30.507151 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":82,"thread_start_us":79,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.125@KRBTEST.COM" created.
I20240507 02:11:30.508039 22159 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.131.125" port: 45921 } has no permanent_uuid. Determining permanent_uuid...
I20240507 02:11:30.508369 22161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:30.508267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.125:45921 (local address 127.0.0.1:60904)
0507 02:11:30.508307 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:30.508345 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.125:45921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20240507 02:11:30.508478 22108 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.131.125:45921: connect: Connection refused (error 111)
I20240507 02:11:30.508859 22161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:30.508798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.125:45921 (local address 127.0.0.1:60916)
0507 02:11:30.508823 (+ 25us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:30.508844 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.125:45921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20240507 02:11:30.508940 22159 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.31.131.125:45921: Network error: Client connection negotiation failed: client connection to 127.31.131.125:45921: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.125.keytab.
Entry for principal kudu/127.31.131.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.125.keytab.
I20240507 02:11:30.519289 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715047604443657-32269-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715047604443657-32269-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715047604443657-32269-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715047604443657-32269-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.125:45921
--webserver_interface=127.31.131.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.125.keytab
--principal=kudu/127.31.131.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:33573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.131.126:39745,127.31.131.125:45921,127.31.131.124:42983
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240507 02:11:30.558158 22159 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.131.125" port: 45921 } attempt: 1
I20240507 02:11:30.558712 22161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:30.558577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.125:45921 (local address 127.0.0.1:60918)
0507 02:11:30.558629 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:30.558676 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.125:45921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20240507 02:11:30.559046 22161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:30.558966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.125:45921 (local address 127.0.0.1:60920)
0507 02:11:30.559002 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:30.559026 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.125:45921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20240507 02:11:30.559142 22159 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.31.131.125:45921: Network error: Client connection negotiation failed: client connection to 127.31.131.125:45921: connect: Connection refused (error 111)
W20240507 02:11:30.595326 22166 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20240507 02:11:30.595595 22166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:11:30.595633 22166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:11:30.595664 22166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:11:30.597280 22166 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240507 02:11:30.597376 22166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:11:30.597410 22166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:11:30.597427 22166 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240507 02:11:30.597440 22166 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240507 02:11:30.598989 22166 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:33573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715047604443657-32269-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715047604443657-32269-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.31.131.126:39745,127.31.131.125:45921,127.31.131.124:42983
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.125.keytab
--principal=kudu/127.31.131.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.125:45921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715047604443657-32269-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.31.131.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715047604443657-32269-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:11:30.599177 22166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:11:30.599346 22166 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:11:30.602953 22172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:11:30.602953 22171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:30.603157 22166 server_base.cc:1034] running on GCE node
W20240507 02:11:30.603190 22174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:30.603395 22166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:11:30.603605 22166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:11:30.604736 22166 hybrid_clock.cc:648] HybridClock initialized: now 1715047890604719 us; error 29 us; skew 500 ppm
May 07 02:11:30 dist-test-slave-v0bg krb5kdc[22078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047890, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:11:30.606997 22166 init.cc:377] Logged in from keytab as kudu/127.31.131.125@KRBTEST.COM (short username kudu)
I20240507 02:11:30.607353 22166 webserver.cc:460] Webserver started at http://127.31.131.125:40975/ using document root <none> and password file <none>
I20240507 02:11:30.607502 22166 fs_manager.cc:359] Metadata directory not provided
I20240507 02:11:30.607537 22166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:11:30.607616 22166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:11:30.608528 22166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715047604443657-32269-0/minicluster-data/master-1/data/instance:
uuid: "6346d184c7d84fcfafb99e03d9b29d13"
format_stamp: "Formatted at 2024-05-07 02:11:30 on dist-test-slave-v0bg"
server_key: "b0f71b1ba51a5e305eeb93003414e2c1"
server_key_iv: "88bc4a327bae7abdc16b10504630ca32"
server_key_version: "encryptionkey@0"
I20240507 02:11:30.608785 22166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715047604443657-32269-0/minicluster-data/master-1/wal/instance:
uuid: "6346d184c7d84fcfafb99e03d9b29d13"
format_stamp: "Formatted at 2024-05-07 02:11:30 on dist-test-slave-v0bg"
server_key: "b0f71b1ba51a5e305eeb93003414e2c1"
server_key_iv: "88bc4a327bae7abdc16b10504630ca32"
server_key_version: "encryptionkey@0"
I20240507 02:11:30.609786 22166 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240507 02:11:30.610369 22181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:30.610523 22166 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240507 02:11:30.610613 22166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715047604443657-32269-0/minicluster-data/master-1/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715047604443657-32269-0/minicluster-data/master-1/wal
uuid: "6346d184c7d84fcfafb99e03d9b29d13"
format_stamp: "Formatted at 2024-05-07 02:11:30 on dist-test-slave-v0bg"
server_key: "b0f71b1ba51a5e305eeb93003414e2c1"
server_key_iv: "88bc4a327bae7abdc16b10504630ca32"
server_key_version: "encryptionkey@0"
I20240507 02:11:30.610666 22166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715047604443657-32269-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715047604443657-32269-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715047604443657-32269-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:11:30.617558 22166 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:11:30.619001 22166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:11:30.619104 22166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:11:30.622195 22166 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.125:45921
I20240507 02:11:30.622236 22233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.125:45921 every 8 connection(s)
I20240507 02:11:30.622502 22166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715047604443657-32269-0/minicluster-data/master-1/data/info.pb
I20240507 02:11:30.622707 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 22166
I20240507 02:11:30.622824 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715047604443657-32269-0/minicluster-data/master-1/wal/instance
I20240507 02:11:30.623000 32269 external_mini_cluster.cc:1428] Setting key 9add31318f30741a74c1b92a1e3ec8eb
I20240507 02:11:30.623422 22234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:11:30.625016 22234 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.131.126" port: 39745 } has no permanent_uuid. Determining permanent_uuid...
I20240507 02:11:30.627295 22159 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.131.125" port: 45921 } attempt: 2
WARNING: no policy specified for kudu/127.31.131.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.124@KRBTEST.COM" created.
May 07 02:11:30 dist-test-slave-v0bg krb5kdc[22078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047890, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.126@KRBTEST.COM for kudu/127.31.131.125@KRBTEST.COM
May 07 02:11:30 dist-test-slave-v0bg krb5kdc[22078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047890, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.125@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:11:30.636206 22161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:30.627507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.125:45921 (local address 127.0.0.1:60936)
0507 02:11:30.627552 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:30.627562 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0507 02:11:30.627597 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:11:30.627818 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:11:30.627819 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:11:30.627962 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:11:30.628096 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:30.628098 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:30.628866 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:11:30.628867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:11:30.629299 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:30.629301 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:30.629369 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:30.629713 (+ 344us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:11:30.629717 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:11:30.632942 (+ 3225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:11:30.635114 (+ 2172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:11:30.635115 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:11:30.635116 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:11:30.635282 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:11:30.635495 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:11:30.635495 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:11:30.635496 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:11:30.635527 (+ 31us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:11:30.635919 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:11:30.635920 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:11:30.635967 (+ 47us) client_negotiation.cc:769] Sending connection context
0507 02:11:30.636014 (+ 47us) client_negotiation.cc:240] Negotiation successful
0507 02:11:30.636103 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20240507 02:11:30.636719 22238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:30.627548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60936 (local address 127.31.131.125:45921)
0507 02:11:30.627698 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:30.627703 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:30.627713 (+ 10us) server_negotiation.cc:407] Connection header received
0507 02:11:30.627729 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:30.627730 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:30.627741 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:30.627803 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:30.628169 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:30.628799 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:30.629381 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:30.629577 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:30.633042 (+ 3465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:30.633051 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:30.633051 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:11:30.633057 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:30.635020 (+ 1963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:30.635369 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:30.635372 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:30.635372 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:30.635423 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:30.635586 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:30.635586 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:30.635587 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:30.635824 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:30.636352 (+ 528us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:30.636433 (+ 81us) server_negotiation.cc:299] Negotiation successful
0507 02:11:30.636545 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":81,"threads_started":1}
I20240507 02:11:30.637045 22162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:30.625632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48900 (local address 127.31.131.126:39745)
0507 02:11:30.625694 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:30.625697 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:30.625898 (+ 201us) server_negotiation.cc:407] Connection header received
0507 02:11:30.626506 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:30.626507 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:30.626517 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:30.626582 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:30.627584 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:30.628291 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:30.628999 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:30.629171 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:30.633818 (+ 4647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:30.633822 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:30.633822 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:11:30.633828 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:30.635893 (+ 2065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:30.636226 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:30.636226 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:30.636227 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:30.636251 (+ 24us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:30.636545 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:30.636546 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:30.636547 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:30.636628 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:30.636690 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:30.636880 (+ 190us) server_negotiation.cc:299] Negotiation successful
0507 02:11:30.636961 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20240507 02:11:30.637377 22159 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.131.124" port: 42983 } has no permanent_uuid. Determining permanent_uuid...
I20240507 02:11:30.637490 22237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:30.625562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:39745 (local address 127.0.0.1:48900)
0507 02:11:30.625778 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:30.625810 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0507 02:11:30.626401 (+ 591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:11:30.626643 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:11:30.626647 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:11:30.627049 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:11:30.627465 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:30.627470 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:30.628402 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:11:30.628403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:11:30.628907 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:30.628909 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:30.629049 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:30.629671 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:11:30.629687 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:11:30.633698 (+ 4011us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:11:30.635975 (+ 2277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:11:30.635981 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:11:30.635983 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:11:30.636158 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:11:30.636415 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:11:30.636416 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:11:30.636417 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:11:30.636451 (+ 34us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:11:30.636687 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:11:30.636693 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:11:30.636743 (+ 50us) client_negotiation.cc:769] Sending connection context
0507 02:11:30.637316 (+ 573us) client_negotiation.cc:240] Negotiation successful
0507 02:11:30.637421 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":81,"threads_started":1}
I20240507 02:11:30.637718 22161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:30.637609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.124:42983 (local address 127.0.0.1:33490)
0507 02:11:30.637658 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:30.637688 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.124:42983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20240507 02:11:30.638047 22161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:30.637962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.124:42983 (local address 127.0.0.1:33502)
0507 02:11:30.637998 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:30.638021 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.124:42983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20240507 02:11:30.638051 22234 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.131.125" port: 45921 } has no permanent_uuid. Determining permanent_uuid...
W20240507 02:11:30.638131 22159 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.31.131.124:42983: Network error: Client connection negotiation failed: client connection to 127.31.131.124:42983: connect: Connection refused (error 111)
May 07 02:11:30 dist-test-slave-v0bg krb5kdc[22078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047890, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.125@KRBTEST.COM for kudu/127.31.131.125@KRBTEST.COM
I20240507 02:11:30.643819 22238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:30.638322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60946 (local address 127.31.131.125:45921)
0507 02:11:30.638374 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:30.638376 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:30.638386 (+ 10us) server_negotiation.cc:407] Connection header received
0507 02:11:30.638454 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:30.638455 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:30.638466 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:30.638515 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:30.638868 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:30.639516 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:30.639995 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:30.640155 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:30.641704 (+ 1549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:30.641707 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:30.641707 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:11:30.641713 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:30.643032 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:30.643296 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:30.643296 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:30.643296 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:30.643312 (+ 16us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:30.643423 (+ 111us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:30.643423 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:30.643423 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:30.643500 (+ 77us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:30.643566 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:30.643669 (+ 103us) server_negotiation.cc:299] Negotiation successful
0507 02:11:30.643732 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20240507 02:11:30.643882 22237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:30.638293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.125:45921 (local address 127.0.0.1:60946)
0507 02:11:30.638354 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:30.638362 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0507 02:11:30.638399 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:11:30.638522 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:11:30.638523 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:11:30.638658 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:11:30.638791 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:30.638792 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:30.639590 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:11:30.639591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:11:30.639897 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:30.639898 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:30.639950 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:30.640323 (+ 373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:11:30.640326 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:11:30.641626 (+ 1300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:11:30.643110 (+ 1484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:11:30.643110 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:11:30.643111 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:11:30.643238 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:11:30.643353 (+ 115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:11:30.643353 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:11:30.643354 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:11:30.643374 (+ 20us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:11:30.643562 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:11:30.643563 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:11:30.643597 (+ 34us) client_negotiation.cc:769] Sending connection context
0507 02:11:30.643653 (+ 56us) client_negotiation.cc:240] Negotiation successful
0507 02:11:30.643787 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":55}
I20240507 02:11:30.644209 22234 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.131.124" port: 42983 } has no permanent_uuid. Determining permanent_uuid...
I20240507 02:11:30.644510 22237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:30.644395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.124:42983 (local address 127.0.0.1:33510)
0507 02:11:30.644453 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:30.644494 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.124:42983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20240507 02:11:30.644869 22183 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.131.124:42983: connect: Connection refused (error 111)
I20240507 02:11:30.645258 22237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:30.645166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.124:42983 (local address 127.0.0.1:33514)
0507 02:11:30.645212 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:30.645236 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.124:42983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20240507 02:11:30.645349 22234 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.31.131.124:42983: Network error: Client connection negotiation failed: client connection to 127.31.131.124:42983: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.124.keytab.
Entry for principal kudu/127.31.131.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.124.keytab.
I20240507 02:11:30.646246 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715047604443657-32269-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715047604443657-32269-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715047604443657-32269-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715047604443657-32269-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.124:42983
--webserver_interface=127.31.131.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.124.keytab
--principal=kudu/127.31.131.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:33573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.131.126:39745,127.31.131.125:45921,127.31.131.124:42983
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240507 02:11:30.681254 22159 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.131.124" port: 42983 } attempt: 1
I20240507 02:11:30.681787 22161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:30.681657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.124:42983 (local address 127.0.0.1:33530)
0507 02:11:30.681716 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:30.681750 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.124:42983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20240507 02:11:30.682128 22161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:30.682045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.124:42983 (local address 127.0.0.1:33540)
0507 02:11:30.682082 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:30.682107 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.124:42983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20240507 02:11:30.682237 22159 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.31.131.124:42983: Network error: Client connection negotiation failed: client connection to 127.31.131.124:42983: connect: Connection refused (error 111)
I20240507 02:11:30.694530 22234 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.131.124" port: 42983 } attempt: 1
I20240507 02:11:30.695006 22237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:30.694883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.124:42983 (local address 127.0.0.1:33550)
0507 02:11:30.694940 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:30.694977 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.124:42983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20240507 02:11:30.695330 22237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:30.695243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.124:42983 (local address 127.0.0.1:33566)
0507 02:11:30.695284 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:30.695309 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.124:42983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20240507 02:11:30.695413 22234 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.31.131.124:42983: Network error: Client connection negotiation failed: client connection to 127.31.131.124:42983: connect: Connection refused (error 111)
W20240507 02:11:30.721262 22241 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20240507 02:11:30.721444 22241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:11:30.721459 22241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:11:30.721479 22241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:11:30.722781 22241 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240507 02:11:30.722815 22241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:11:30.722827 22241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:11:30.722838 22241 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240507 02:11:30.722862 22241 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240507 02:11:30.724275 22241 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:33573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715047604443657-32269-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715047604443657-32269-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.31.131.126:39745,127.31.131.125:45921,127.31.131.124:42983
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.124.keytab
--principal=kudu/127.31.131.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.124:42983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715047604443657-32269-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.31.131.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22241
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715047604443657-32269-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:11:30.724454 22241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:11:30.724619 22241 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:11:30.728194 22249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:30.728336 22241 server_base.cc:1034] running on GCE node
W20240507 02:11:30.728245 22246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:11:30.728430 22247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:30.728646 22241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:11:30.728852 22241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:11:30.729395 22159 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.131.124" port: 42983 } attempt: 2
I20240507 02:11:30.729768 22161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:30.729660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.124:42983 (local address 127.0.0.1:33582)
0507 02:11:30.729712 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:30.729740 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.124:42983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20240507 02:11:30.729974 22241 hybrid_clock.cc:648] HybridClock initialized: now 1715047890729956 us; error 30 us; skew 500 ppm
I20240507 02:11:30.730077 22161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:30.729983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.124:42983 (local address 127.0.0.1:33594)
0507 02:11:30.730026 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:30.730051 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.124:42983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20240507 02:11:30.730185 22159 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.31.131.124:42983: Network error: Client connection negotiation failed: client connection to 127.31.131.124:42983: connect: Connection refused (error 111)
May 07 02:11:30 dist-test-slave-v0bg krb5kdc[22078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047890, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:11:30.732249 22241 init.cc:377] Logged in from keytab as kudu/127.31.131.124@KRBTEST.COM (short username kudu)
I20240507 02:11:30.732627 22241 webserver.cc:460] Webserver started at http://127.31.131.124:34901/ using document root <none> and password file <none>
I20240507 02:11:30.732779 22241 fs_manager.cc:359] Metadata directory not provided
I20240507 02:11:30.732813 22241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:11:30.732910 22241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:11:30.733829 22241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715047604443657-32269-0/minicluster-data/master-2/data/instance:
uuid: "ef114a74cfa34db29288c61ea16c85a6"
format_stamp: "Formatted at 2024-05-07 02:11:30 on dist-test-slave-v0bg"
server_key: "ccf2cfad352bd06d8969d63dfe1d1526"
server_key_iv: "3cdb1a0c387497b11cf6daba75e2a118"
server_key_version: "encryptionkey@0"
I20240507 02:11:30.734117 22241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715047604443657-32269-0/minicluster-data/master-2/wal/instance:
uuid: "ef114a74cfa34db29288c61ea16c85a6"
format_stamp: "Formatted at 2024-05-07 02:11:30 on dist-test-slave-v0bg"
server_key: "ccf2cfad352bd06d8969d63dfe1d1526"
server_key_iv: "3cdb1a0c387497b11cf6daba75e2a118"
server_key_version: "encryptionkey@0"
I20240507 02:11:30.735210 22241 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240507 02:11:30.735788 22256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:30.735950 22241 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240507 02:11:30.736016 22241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715047604443657-32269-0/minicluster-data/master-2/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715047604443657-32269-0/minicluster-data/master-2/wal
uuid: "ef114a74cfa34db29288c61ea16c85a6"
format_stamp: "Formatted at 2024-05-07 02:11:30 on dist-test-slave-v0bg"
server_key: "ccf2cfad352bd06d8969d63dfe1d1526"
server_key_iv: "3cdb1a0c387497b11cf6daba75e2a118"
server_key_version: "encryptionkey@0"
I20240507 02:11:30.736060 22241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715047604443657-32269-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715047604443657-32269-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715047604443657-32269-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:11:30.754359 22241 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:11:30.756316 22241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:11:30.756415 22241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:11:30.759943 22241 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.124:42983
I20240507 02:11:30.759968 22308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.124:42983 every 8 connection(s)
I20240507 02:11:30.760263 22241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715047604443657-32269-0/minicluster-data/master-2/data/info.pb
I20240507 02:11:30.761024 22309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:11:30.762377 22309 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.131.126" port: 39745 } has no permanent_uuid. Determining permanent_uuid...
I20240507 02:11:30.763536 22234 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.131.124" port: 42983 } attempt: 2
May 07 02:11:30 dist-test-slave-v0bg krb5kdc[22078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047890, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.124@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
May 07 02:11:30 dist-test-slave-v0bg krb5kdc[22078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047890, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.125@KRBTEST.COM for kudu/127.31.131.124@KRBTEST.COM
I20240507 02:11:30.769915 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 22241
I20240507 02:11:30.770020 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1715047604443657-32269-0/minicluster-data/master-2/wal/instance
I20240507 02:11:30.770179 32269 external_mini_cluster.cc:1428] Setting key e6d8e5871f01fa47a343fc17d4373f0c
I20240507 02:11:30.770771 22311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:30.762984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:39745 (local address 127.0.0.1:48908)
0507 02:11:30.763121 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:30.763149 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0507 02:11:30.763589 (+ 440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:11:30.763967 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:11:30.763970 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:11:30.764244 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:11:30.764465 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:30.764469 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:30.765290 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:11:30.765290 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:11:30.765697 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:30.765699 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:30.765822 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:30.766307 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:11:30.766327 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:11:30.767794 (+ 1467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:11:30.769329 (+ 1535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:11:30.769334 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:11:30.769337 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:11:30.769602 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:11:30.769735 (+ 133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:11:30.769736 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:11:30.769736 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:11:30.769823 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:11:30.770066 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:11:30.770071 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:11:30.770189 (+ 118us) client_negotiation.cc:769] Sending connection context
0507 02:11:30.770331 (+ 142us) client_negotiation.cc:240] Negotiation successful
0507 02:11:30.770630 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":76,"threads_started":1}
I20240507 02:11:30.770814 22162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:30.763027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48908 (local address 127.31.131.126:39745)
0507 02:11:30.763087 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:30.763088 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:30.763221 (+ 133us) server_negotiation.cc:407] Connection header received
0507 02:11:30.763831 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:30.763833 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:30.763844 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:30.763899 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:30.764550 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:30.765224 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:30.765765 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:30.765935 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:30.767884 (+ 1949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:30.767888 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:30.767889 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:11:30.767895 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:30.769178 (+ 1283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:30.769669 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:30.769669 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:30.769670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:30.769685 (+ 15us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:30.769873 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:30.769875 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:30.769875 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:30.769980 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:30.770041 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:30.770636 (+ 595us) server_negotiation.cc:299] Negotiation successful
0507 02:11:30.770718 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20240507 02:11:30.770814 22237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:30.763763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.124:42983 (local address 127.0.0.1:33608)
0507 02:11:30.763809 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:30.763819 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0507 02:11:30.763858 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:11:30.764115 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:11:30.764116 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:11:30.764272 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:11:30.764402 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:30.764404 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:30.765552 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:11:30.765553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:11:30.765913 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:30.765915 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:30.765981 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:30.766486 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:11:30.766490 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:11:30.768071 (+ 1581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:11:30.769839 (+ 1768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:11:30.769840 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:11:30.769841 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:11:30.770022 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:11:30.770213 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:11:30.770214 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:11:30.770214 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:11:30.770247 (+ 33us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:11:30.770494 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:11:30.770495 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:11:30.770544 (+ 49us) client_negotiation.cc:769] Sending connection context
0507 02:11:30.770638 (+ 94us) client_negotiation.cc:240] Negotiation successful
0507 02:11:30.770720 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20240507 02:11:30.771222 22312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:30.763803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33608 (local address 127.31.131.124:42983)
0507 02:11:30.763989 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:30.763996 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:30.764006 (+ 10us) server_negotiation.cc:407] Connection header received
0507 02:11:30.764022 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:30.764024 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:30.764038 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:30.764103 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:30.764478 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:30.765480 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:30.765987 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:30.766159 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:30.768142 (+ 1983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:30.768150 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:30.768150 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:11:30.768155 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:30.769762 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:30.770108 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:30.770110 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:30.770111 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:30.770145 (+ 34us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:30.770310 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:30.770311 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:30.770311 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:30.770414 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:30.770502 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:30.771059 (+ 557us) server_negotiation.cc:299] Negotiation successful
0507 02:11:30.771143 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":69,"threads_started":1}
I20240507 02:11:30.771240 22309 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.131.125" port: 45921 } has no permanent_uuid. Determining permanent_uuid...
I20240507 02:11:30.772578 22234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6346d184c7d84fcfafb99e03d9b29d13: Bootstrap starting.
I20240507 02:11:30.773152 22234 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6346d184c7d84fcfafb99e03d9b29d13: Neither blocks nor log segments found. Creating new log.
I20240507 02:11:30.773411 22234 log.cc:824] T 00000000000000000000000000000000 P 6346d184c7d84fcfafb99e03d9b29d13: Log is configured to *not* fsync() on all Append() calls
I20240507 02:11:30.774214 22234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6346d184c7d84fcfafb99e03d9b29d13: No bootstrap required, opened a new log
May 07 02:11:30 dist-test-slave-v0bg krb5kdc[22078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047890, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.124@KRBTEST.COM for kudu/127.31.131.125@KRBTEST.COM
May 07 02:11:30 dist-test-slave-v0bg krb5kdc[22078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.131.124@KRBTEST.COM
I20240507 02:11:30.775928 22234 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6346d184c7d84fcfafb99e03d9b29d13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09beb00ac8bf4b96a5304facaf05b0b7" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 39745 } } peers { permanent_uuid: "6346d184c7d84fcfafb99e03d9b29d13" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 45921 } } peers { permanent_uuid: "ef114a74cfa34db29288c61ea16c85a6" member_type: VOTER last_known_addr { host: "127.31.131.124" port: 42983 } }
I20240507 02:11:30.776047 22234 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6346d184c7d84fcfafb99e03d9b29d13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6346d184c7d84fcfafb99e03d9b29d13, State: Initialized, Role: FOLLOWER
I20240507 02:11:30.776108 22234 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6346d184c7d84fcfafb99e03d9b29d13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09beb00ac8bf4b96a5304facaf05b0b7" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 39745 } } peers { permanent_uuid: "6346d184c7d84fcfafb99e03d9b29d13" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 45921 } } peers { permanent_uuid: "ef114a74cfa34db29288c61ea16c85a6" member_type: VOTER last_known_addr { host: "127.31.131.124" port: 42983 } }
I20240507 02:11:30.776381 22234 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6346d184c7d84fcfafb99e03d9b29d13 [sys.catalog]: configured and running, proceeding with master startup.
I20240507 02:11:30.777233 22316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6346d184c7d84fcfafb99e03d9b29d13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09beb00ac8bf4b96a5304facaf05b0b7" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 39745 } } peers { permanent_uuid: "6346d184c7d84fcfafb99e03d9b29d13" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 45921 } } peers { permanent_uuid: "ef114a74cfa34db29288c61ea16c85a6" member_type: VOTER last_known_addr { host: "127.31.131.124" port: 42983 } } }
I20240507 02:11:30.777417 22311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:30.771515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.125:45921 (local address 127.0.0.1:60958)
0507 02:11:30.771627 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:30.771635 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0507 02:11:30.771679 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:11:30.771770 (+ 91us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:11:30.771770 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:11:30.771896 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:11:30.772088 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:30.772090 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:30.772831 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:11:30.772832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:11:30.773298 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:30.773299 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:30.773359 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:30.773754 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:11:30.773759 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:11:30.774911 (+ 1152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:11:30.776555 (+ 1644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:11:30.776556 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:11:30.776556 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:11:30.776722 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:11:30.776873 (+ 151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:11:30.776874 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:11:30.776874 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:11:30.776900 (+ 26us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:11:30.777166 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:11:30.777167 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:11:30.777206 (+ 39us) client_negotiation.cc:769] Sending connection context
0507 02:11:30.777263 (+ 57us) client_negotiation.cc:240] Negotiation successful
0507 02:11:30.777340 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100}
I20240507 02:11:30.777678 22238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:30.771569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60958 (local address 127.31.131.125:45921)
0507 02:11:30.771604 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:30.771605 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:30.771685 (+ 80us) server_negotiation.cc:407] Connection header received
0507 02:11:30.771711 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:30.771712 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:30.771722 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:30.771766 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:30.772166 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:30.772759 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:30.773369 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:30.774054 (+ 685us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:30.774981 (+ 927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:30.774984 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:30.774984 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:11:30.774990 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:30.776468 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:30.776791 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:30.776792 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:30.776792 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:30.776814 (+ 22us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:30.776965 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:30.776967 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:30.776967 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:30.777086 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:30.777142 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:30.777518 (+ 376us) server_negotiation.cc:299] Negotiation successful
0507 02:11:30.777596 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20240507 02:11:30.778311 22309 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.131.124" port: 42983 } has no permanent_uuid. Determining permanent_uuid...
I20240507 02:11:30.778851 22316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6346d184c7d84fcfafb99e03d9b29d13 [sys.catalog]: This master's current role is: FOLLOWER
I20240507 02:11:30.779397 22313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:30.770476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33616 (local address 127.31.131.124:42983)
0507 02:11:30.770606 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:30.770608 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:30.771504 (+ 896us) server_negotiation.cc:407] Connection header received
0507 02:11:30.771521 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:30.771522 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:30.771530 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:30.771582 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:30.772430 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:30.772992 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:30.773807 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:30.773992 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:30.775908 (+ 1916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:30.775912 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:30.775912 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:11:30.775917 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:30.778009 (+ 2092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:30.778762 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:30.778763 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:30.778763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:30.778785 (+ 22us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:30.778964 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:30.778965 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:30.778965 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:30.779042 (+ 77us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:30.779143 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:30.779236 (+ 93us) server_negotiation.cc:299] Negotiation successful
0507 02:11:30.779306 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":93,"threads_started":1}
I20240507 02:11:30.778359 22166 master_runner.cc:186] Error getting master registration for 127.31.131.126:39745: OK, instance_id { permanent_uuid: "09beb00ac8bf4b96a5304facaf05b0b7" instance_seqno: 1715047890494941 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20240507 02:11:30.780267 22166 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240507 02:11:30.778354 22327 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6346d184c7d84fcfafb99e03d9b29d13: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240507 02:11:30.780687 22327 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
May 07 02:11:30 dist-test-slave-v0bg krb5kdc[22078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
May 07 02:11:30 dist-test-slave-v0bg krb5kdc[22078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047890, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.124@KRBTEST.COM for kudu/127.31.131.124@KRBTEST.COM
I20240507 02:11:30.786393 22311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:30.780436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.124:42983 (local address 127.0.0.1:33618)
0507 02:11:30.780491 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:30.780499 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0507 02:11:30.780531 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:11:30.780762 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:11:30.780762 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:11:30.780886 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:11:30.781026 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:30.781027 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:30.782107 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:11:30.782109 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:11:30.782632 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:30.782634 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:30.782700 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:30.783075 (+ 375us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:11:30.783079 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:11:30.784077 (+ 998us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:11:30.785625 (+ 1548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:11:30.785626 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:11:30.785626 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:11:30.785748 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:11:30.785889 (+ 141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:11:30.785889 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:11:30.785890 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:11:30.785915 (+ 25us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:11:30.786143 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:11:30.786144 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:11:30.786174 (+ 30us) client_negotiation.cc:769] Sending connection context
0507 02:11:30.786232 (+ 58us) client_negotiation.cc:240] Negotiation successful
0507 02:11:30.786308 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20240507 02:11:30.786520 22162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:30.779866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48920 (local address 127.31.131.126:39745)
0507 02:11:30.779898 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:30.779900 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:30.779939 (+ 39us) server_negotiation.cc:407] Connection header received
0507 02:11:30.779997 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:30.779998 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:30.780007 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:30.780045 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:30.780613 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:30.781207 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:30.781663 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:30.781813 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:30.784229 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:30.784232 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:30.784233 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:11:30.784238 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:30.785555 (+ 1317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:30.785982 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:30.785983 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:30.785984 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:30.786003 (+ 19us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:30.786116 (+ 113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:30.786116 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:30.786117 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:30.786191 (+ 74us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:30.786304 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:30.786359 (+ 55us) server_negotiation.cc:299] Negotiation successful
0507 02:11:30.786436 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20240507 02:11:30.786747 22313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:30.780631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33618 (local address 127.31.131.124:42983)
0507 02:11:30.780663 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:30.780665 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:30.780670 (+ 5us) server_negotiation.cc:407] Connection header received
0507 02:11:30.780685 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:30.780686 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:30.780694 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:30.780735 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:30.781104 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:30.781740 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:30.782703 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:30.783161 (+ 458us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:30.784154 (+ 993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:30.784157 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:30.784157 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:11:30.784164 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:30.785550 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:30.785805 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:30.785806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:30.785806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:30.785827 (+ 21us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:30.785971 (+ 144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:30.785971 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:30.785972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:30.786077 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:30.786125 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:30.786603 (+ 478us) server_negotiation.cc:299] Negotiation successful
0507 02:11:30.786674 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20240507 02:11:30.788166 22309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef114a74cfa34db29288c61ea16c85a6: Bootstrap starting.
I20240507 02:11:30.788753 22309 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef114a74cfa34db29288c61ea16c85a6: Neither blocks nor log segments found. Creating new log.
I20240507 02:11:30.789013 22309 log.cc:824] T 00000000000000000000000000000000 P ef114a74cfa34db29288c61ea16c85a6: Log is configured to *not* fsync() on all Append() calls
I20240507 02:11:30.789824 22309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef114a74cfa34db29288c61ea16c85a6: No bootstrap required, opened a new log
May 07 02:11:30 dist-test-slave-v0bg krb5kdc[22078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.131.125@KRBTEST.COM
I20240507 02:11:30.791563 22309 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef114a74cfa34db29288c61ea16c85a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09beb00ac8bf4b96a5304facaf05b0b7" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 39745 } } peers { permanent_uuid: "6346d184c7d84fcfafb99e03d9b29d13" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 45921 } } peers { permanent_uuid: "ef114a74cfa34db29288c61ea16c85a6" member_type: VOTER last_known_addr { host: "127.31.131.124" port: 42983 } }
I20240507 02:11:30.791743 22309 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef114a74cfa34db29288c61ea16c85a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef114a74cfa34db29288c61ea16c85a6, State: Initialized, Role: FOLLOWER
I20240507 02:11:30.791822 22309 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef114a74cfa34db29288c61ea16c85a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09beb00ac8bf4b96a5304facaf05b0b7" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 39745 } } peers { permanent_uuid: "6346d184c7d84fcfafb99e03d9b29d13" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 45921 } } peers { permanent_uuid: "ef114a74cfa34db29288c61ea16c85a6" member_type: VOTER last_known_addr { host: "127.31.131.124" port: 42983 } }
I20240507 02:11:30.792176 22309 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef114a74cfa34db29288c61ea16c85a6 [sys.catalog]: configured and running, proceeding with master startup.
I20240507 02:11:30.792143 22330 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef114a74cfa34db29288c61ea16c85a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09beb00ac8bf4b96a5304facaf05b0b7" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 39745 } } peers { permanent_uuid: "6346d184c7d84fcfafb99e03d9b29d13" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 45921 } } peers { permanent_uuid: "ef114a74cfa34db29288c61ea16c85a6" member_type: VOTER last_known_addr { host: "127.31.131.124" port: 42983 } } }
I20240507 02:11:30.792318 22330 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef114a74cfa34db29288c61ea16c85a6 [sys.catalog]: This master's current role is: FOLLOWER
I20240507 02:11:30.792392 22238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:30.786982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60968 (local address 127.31.131.125:45921)
0507 02:11:30.787016 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:30.787018 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:30.787025 (+ 7us) server_negotiation.cc:407] Connection header received
0507 02:11:30.787066 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:30.787066 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:30.787075 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:30.787137 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:30.787613 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:30.788013 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:30.788443 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:30.788596 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:30.790299 (+ 1703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:30.790302 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:30.790303 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:11:30.790309 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:30.791451 (+ 1142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:30.791696 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:30.791697 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:30.791698 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:30.791716 (+ 18us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:30.791861 (+ 145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:30.791861 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:30.791862 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:30.791947 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:30.792053 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:30.792236 (+ 183us) server_negotiation.cc:299] Negotiation successful
0507 02:11:30.792312 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20240507 02:11:30.793663 32269 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240507 02:11:30.794423 22133 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "09beb00ac8bf4b96a5304facaf05b0b7"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48920
W20240507 02:11:30.795589 22341 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ef114a74cfa34db29288c61ea16c85a6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240507 02:11:30.795696 22341 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240507 02:11:30.794103 22241 master_runner.cc:186] Error getting master registration for 127.31.131.126:39745: OK, instance_id { permanent_uuid: "09beb00ac8bf4b96a5304facaf05b0b7" instance_seqno: 1715047890494941 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20240507 02:11:30.796047 22241 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240507 02:11:30.837309 22159 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.131.124" port: 42983 } attempt: 3
May 07 02:11:30 dist-test-slave-v0bg krb5kdc[22078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047890, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.126@KRBTEST.COM for kudu/127.31.131.124@KRBTEST.COM
I20240507 02:11:30.845017 22161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:30.837737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.124:42983 (local address 127.0.0.1:33626)
0507 02:11:30.837855 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:30.837866 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0507 02:11:30.837910 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:11:30.838141 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:11:30.838142 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:11:30.838357 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:11:30.838607 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:30.838609 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:30.839569 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:11:30.839570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:11:30.840119 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:30.840121 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:30.840196 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:30.840819 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:11:30.840824 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:11:30.842331 (+ 1507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:11:30.844132 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:11:30.844133 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:11:30.844133 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:11:30.844310 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:11:30.844479 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:11:30.844480 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:11:30.844480 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:11:30.844509 (+ 29us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:11:30.844715 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:11:30.844716 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:11:30.844761 (+ 45us) client_negotiation.cc:769] Sending connection context
0507 02:11:30.844821 (+ 60us) client_negotiation.cc:240] Negotiation successful
0507 02:11:30.844911 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97}
I20240507 02:11:30.845358 22313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:30.837812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33626 (local address 127.31.131.124:42983)
0507 02:11:30.837993 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:30.837995 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:30.838004 (+ 9us) server_negotiation.cc:407] Connection header received
0507 02:11:30.838030 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:30.838031 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:30.838042 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:30.838086 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:30.838697 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:30.839494 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:30.840194 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:30.840367 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:30.842415 (+ 2048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:30.842420 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:30.842420 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:11:30.842427 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:30.844051 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:30.844395 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:30.844397 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:30.844397 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:30.844421 (+ 24us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:30.844558 (+ 137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:30.844559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:30.844559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:30.844663 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:30.844720 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:30.845184 (+ 464us) server_negotiation.cc:299] Negotiation successful
0507 02:11:30.845267 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160}
I20240507 02:11:30.846889 22159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09beb00ac8bf4b96a5304facaf05b0b7: Bootstrap starting.
I20240507 02:11:30.847546 22159 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09beb00ac8bf4b96a5304facaf05b0b7: Neither blocks nor log segments found. Creating new log.
I20240507 02:11:30.847803 22159 log.cc:824] T 00000000000000000000000000000000 P 09beb00ac8bf4b96a5304facaf05b0b7: Log is configured to *not* fsync() on all Append() calls
I20240507 02:11:30.848671 22159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09beb00ac8bf4b96a5304facaf05b0b7: No bootstrap required, opened a new log
I20240507 02:11:30.849838 22159 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09beb00ac8bf4b96a5304facaf05b0b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09beb00ac8bf4b96a5304facaf05b0b7" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 39745 } } peers { permanent_uuid: "6346d184c7d84fcfafb99e03d9b29d13" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 45921 } } peers { permanent_uuid: "ef114a74cfa34db29288c61ea16c85a6" member_type: VOTER last_known_addr { host: "127.31.131.124" port: 42983 } }
I20240507 02:11:30.849977 22159 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09beb00ac8bf4b96a5304facaf05b0b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09beb00ac8bf4b96a5304facaf05b0b7, State: Initialized, Role: FOLLOWER
I20240507 02:11:30.850034 22159 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09beb00ac8bf4b96a5304facaf05b0b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09beb00ac8bf4b96a5304facaf05b0b7" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 39745 } } peers { permanent_uuid: "6346d184c7d84fcfafb99e03d9b29d13" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 45921 } } peers { permanent_uuid: "ef114a74cfa34db29288c61ea16c85a6" member_type: VOTER last_known_addr { host: "127.31.131.124" port: 42983 } }
I20240507 02:11:30.850322 22159 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09beb00ac8bf4b96a5304facaf05b0b7 [sys.catalog]: configured and running, proceeding with master startup.
I20240507 02:11:30.850296 22344 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09beb00ac8bf4b96a5304facaf05b0b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09beb00ac8bf4b96a5304facaf05b0b7" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 39745 } } peers { permanent_uuid: "6346d184c7d84fcfafb99e03d9b29d13" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 45921 } } peers { permanent_uuid: "ef114a74cfa34db29288c61ea16c85a6" member_type: VOTER last_known_addr { host: "127.31.131.124" port: 42983 } } }
I20240507 02:11:30.850378 22344 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09beb00ac8bf4b96a5304facaf05b0b7 [sys.catalog]: This master's current role is: FOLLOWER
I20240507 02:11:30.850665 22133 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09beb00ac8bf4b96a5304facaf05b0b7 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20240507 02:11:30.850726 22133 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09beb00ac8bf4b96a5304facaf05b0b7 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:11:30.851529 22133 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09beb00ac8bf4b96a5304facaf05b0b7 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09beb00ac8bf4b96a5304facaf05b0b7" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 39745 } } peers { permanent_uuid: "6346d184c7d84fcfafb99e03d9b29d13" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 45921 } } peers { permanent_uuid: "ef114a74cfa34db29288c61ea16c85a6" member_type: VOTER last_known_addr { host: "127.31.131.124" port: 42983 } }
I20240507 02:11:30.851747 22133 leader_election.cc:288] T 00000000000000000000000000000000 P 09beb00ac8bf4b96a5304facaf05b0b7 [CANDIDATE]: Term 1 election: Requested vote from peers 6346d184c7d84fcfafb99e03d9b29d13 (127.31.131.125:45921), ef114a74cfa34db29288c61ea16c85a6 (127.31.131.124:42983)
I20240507 02:11:30.852066 22208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "09beb00ac8bf4b96a5304facaf05b0b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6346d184c7d84fcfafb99e03d9b29d13"
I20240507 02:11:30.852133 22283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "09beb00ac8bf4b96a5304facaf05b0b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ef114a74cfa34db29288c61ea16c85a6"
I20240507 02:11:30.852190 22208 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6346d184c7d84fcfafb99e03d9b29d13 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:11:30.852237 22283 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef114a74cfa34db29288c61ea16c85a6 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:11:30.853041 22208 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6346d184c7d84fcfafb99e03d9b29d13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09beb00ac8bf4b96a5304facaf05b0b7 in term 1.
I20240507 02:11:30.853080 22283 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ef114a74cfa34db29288c61ea16c85a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09beb00ac8bf4b96a5304facaf05b0b7 in term 1.
I20240507 02:11:30.853245 22108 leader_election.cc:302] T 00000000000000000000000000000000 P 09beb00ac8bf4b96a5304facaf05b0b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09beb00ac8bf4b96a5304facaf05b0b7, 6346d184c7d84fcfafb99e03d9b29d13; no voters:
I20240507 02:11:30.854329 22344 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09beb00ac8bf4b96a5304facaf05b0b7 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:11:30.854817 22344 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09beb00ac8bf4b96a5304facaf05b0b7 [term 1 LEADER]: Becoming Leader. State: Replica: 09beb00ac8bf4b96a5304facaf05b0b7, State: Running, Role: LEADER
I20240507 02:11:30.854971 22344 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09beb00ac8bf4b96a5304facaf05b0b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09beb00ac8bf4b96a5304facaf05b0b7" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 39745 } } peers { permanent_uuid: "6346d184c7d84fcfafb99e03d9b29d13" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 45921 } } peers { permanent_uuid: "ef114a74cfa34db29288c61ea16c85a6" member_type: VOTER last_known_addr { host: "127.31.131.124" port: 42983 } }
I20240507 02:11:30.855413 22360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09beb00ac8bf4b96a5304facaf05b0b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09beb00ac8bf4b96a5304facaf05b0b7. Latest consensus state: current_term: 1 leader_uuid: "09beb00ac8bf4b96a5304facaf05b0b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09beb00ac8bf4b96a5304facaf05b0b7" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 39745 } } peers { permanent_uuid: "6346d184c7d84fcfafb99e03d9b29d13" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 45921 } } peers { permanent_uuid: "ef114a74cfa34db29288c61ea16c85a6" member_type: VOTER last_known_addr { host: "127.31.131.124" port: 42983 } } }
I20240507 02:11:30.855504 22360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09beb00ac8bf4b96a5304facaf05b0b7 [sys.catalog]: This master's current role is: LEADER
W20240507 02:11:30.856482 22355 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 09beb00ac8bf4b96a5304facaf05b0b7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240507 02:11:30.856528 22355 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240507 02:11:30.857064 22362 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 02:11:30.857225 22362 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240507 02:11:30.858541 22283 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ef114a74cfa34db29288c61ea16c85a6 [term 1 FOLLOWER]: Refusing update from remote peer 09beb00ac8bf4b96a5304facaf05b0b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 02:11:30.858580 22208 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6346d184c7d84fcfafb99e03d9b29d13 [term 1 FOLLOWER]: Refusing update from remote peer 09beb00ac8bf4b96a5304facaf05b0b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 02:11:30.858830 22344 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 09beb00ac8bf4b96a5304facaf05b0b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef114a74cfa34db29288c61ea16c85a6" member_type: VOTER last_known_addr { host: "127.31.131.124" port: 42983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 02:11:30.858970 22360 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 09beb00ac8bf4b96a5304facaf05b0b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6346d184c7d84fcfafb99e03d9b29d13" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 45921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 02:11:30.859863 22330 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef114a74cfa34db29288c61ea16c85a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09beb00ac8bf4b96a5304facaf05b0b7. Latest consensus state: current_term: 1 leader_uuid: "09beb00ac8bf4b96a5304facaf05b0b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09beb00ac8bf4b96a5304facaf05b0b7" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 39745 } } peers { permanent_uuid: "6346d184c7d84fcfafb99e03d9b29d13" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 45921 } } peers { permanent_uuid: "ef114a74cfa34db29288c61ea16c85a6" member_type: VOTER last_known_addr { host: "127.31.131.124" port: 42983 } } }
I20240507 02:11:30.859951 22330 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef114a74cfa34db29288c61ea16c85a6 [sys.catalog]: This master's current role is: FOLLOWER
I20240507 02:11:30.859984 22316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6346d184c7d84fcfafb99e03d9b29d13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09beb00ac8bf4b96a5304facaf05b0b7. Latest consensus state: current_term: 1 leader_uuid: "09beb00ac8bf4b96a5304facaf05b0b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09beb00ac8bf4b96a5304facaf05b0b7" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 39745 } } peers { permanent_uuid: "6346d184c7d84fcfafb99e03d9b29d13" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 45921 } } peers { permanent_uuid: "ef114a74cfa34db29288c61ea16c85a6" member_type: VOTER last_known_addr { host: "127.31.131.124" port: 42983 } } }
I20240507 02:11:30.860054 22316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6346d184c7d84fcfafb99e03d9b29d13 [sys.catalog]: This master's current role is: FOLLOWER
I20240507 02:11:30.860870 22360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09beb00ac8bf4b96a5304facaf05b0b7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "09beb00ac8bf4b96a5304facaf05b0b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09beb00ac8bf4b96a5304facaf05b0b7" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 39745 } } peers { permanent_uuid: "6346d184c7d84fcfafb99e03d9b29d13" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 45921 } } peers { permanent_uuid: "ef114a74cfa34db29288c61ea16c85a6" member_type: VOTER last_known_addr { host: "127.31.131.124" port: 42983 } } }
I20240507 02:11:30.860949 22360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09beb00ac8bf4b96a5304facaf05b0b7 [sys.catalog]: This master's current role is: LEADER
I20240507 02:11:30.861245 22316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6346d184c7d84fcfafb99e03d9b29d13 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "09beb00ac8bf4b96a5304facaf05b0b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09beb00ac8bf4b96a5304facaf05b0b7" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 39745 } } peers { permanent_uuid: "6346d184c7d84fcfafb99e03d9b29d13" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 45921 } } peers { permanent_uuid: "ef114a74cfa34db29288c61ea16c85a6" member_type: VOTER last_known_addr { host: "127.31.131.124" port: 42983 } } }
I20240507 02:11:30.861310 22316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6346d184c7d84fcfafb99e03d9b29d13 [sys.catalog]: This master's current role is: FOLLOWER
I20240507 02:11:30.861570 22367 mvcc.cc:205] Tried to move back new op lower bound from 7024836160954548224 to 7024836160942755840. Current Snapshot: MvccSnapshot[applied={T|T < 7024836160954548224}]
I20240507 02:11:30.862020 22369 mvcc.cc:205] Tried to move back new op lower bound from 7024836160954548224 to 7024836160942755840. Current Snapshot: MvccSnapshot[applied={T|T < 7024836160954548224}]
I20240507 02:11:30.866034 22362 catalog_manager.cc:1347] Generated new cluster ID: cff42734dc304863824edc5cc36204cc
I20240507 02:11:30.866094 22362 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240507 02:11:30.870637 22344 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09beb00ac8bf4b96a5304facaf05b0b7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "09beb00ac8bf4b96a5304facaf05b0b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09beb00ac8bf4b96a5304facaf05b0b7" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 39745 } } peers { permanent_uuid: "6346d184c7d84fcfafb99e03d9b29d13" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 45921 } } peers { permanent_uuid: "ef114a74cfa34db29288c61ea16c85a6" member_type: VOTER last_known_addr { host: "127.31.131.124" port: 42983 } } }
I20240507 02:11:30.870739 22344 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09beb00ac8bf4b96a5304facaf05b0b7 [sys.catalog]: This master's current role is: LEADER
I20240507 02:11:30.871289 22330 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef114a74cfa34db29288c61ea16c85a6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "09beb00ac8bf4b96a5304facaf05b0b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09beb00ac8bf4b96a5304facaf05b0b7" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 39745 } } peers { permanent_uuid: "6346d184c7d84fcfafb99e03d9b29d13" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 45921 } } peers { permanent_uuid: "ef114a74cfa34db29288c61ea16c85a6" member_type: VOTER last_known_addr { host: "127.31.131.124" port: 42983 } } }
I20240507 02:11:30.871372 22330 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef114a74cfa34db29288c61ea16c85a6 [sys.catalog]: This master's current role is: FOLLOWER
I20240507 02:11:30.875159 22162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:30.855993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48922 (local address 127.31.131.126:39745)
0507 02:11:30.856025 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:30.856026 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:30.857978 (+ 1952us) server_negotiation.cc:407] Connection header received
0507 02:11:30.858071 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:30.858072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:30.858081 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:30.858108 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:30.867396 (+ 9288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:30.868040 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:30.868698 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:30.868842 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:30.871862 (+ 3020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:30.871865 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:30.871865 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:11:30.871871 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:30.873968 (+ 2097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:30.874319 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:30.874320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:30.874321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:30.874347 (+ 26us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:30.874488 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:30.874488 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:30.874489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:30.874601 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:30.874678 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:30.874974 (+ 296us) server_negotiation.cc:299] Negotiation successful
0507 02:11:30.875070 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20240507 02:11:30.876199 22313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:30.856907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33640 (local address 127.31.131.124:42983)
0507 02:11:30.857017 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:30.857019 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:30.866903 (+ 9884us) server_negotiation.cc:407] Connection header received
0507 02:11:30.867008 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:30.867010 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:30.867020 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:30.867050 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:30.867961 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:30.868607 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:30.869322 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:30.869482 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:30.871761 (+ 2279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:30.871764 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:30.871764 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:11:30.871770 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:30.874822 (+ 3052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:30.875171 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:30.875173 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:30.875173 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:30.875198 (+ 25us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:30.875542 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:30.875543 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:30.875543 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:30.875646 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:30.875741 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:30.876026 (+ 285us) server_negotiation.cc:299] Negotiation successful
0507 02:11:30.876120 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20240507 02:11:30.879498 22238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:30.855799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60978 (local address 127.31.131.125:45921)
0507 02:11:30.855839 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:30.855841 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:30.872810 (+ 16969us) server_negotiation.cc:407] Connection header received
0507 02:11:30.872835 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:30.872836 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:30.872846 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:30.872898 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:30.873629 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:30.874296 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:30.875033 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:30.875191 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:30.876837 (+ 1646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:30.876840 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:30.876841 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:11:30.876847 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:30.878360 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:30.878733 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:30.878734 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:30.878734 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:30.878760 (+ 26us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:30.878915 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:30.878927 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:30.878928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:30.879048 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:30.879124 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:30.879334 (+ 210us) server_negotiation.cc:299] Negotiation successful
0507 02:11:30.879411 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20240507 02:11:30.902081 22362 catalog_manager.cc:1370] Generated new certificate authority record
I20240507 02:11:30.903384 22362 catalog_manager.cc:1504] Loading token signing keys...
I20240507 02:11:30.919342 22362 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 09beb00ac8bf4b96a5304facaf05b0b7: Generated new TSK 0
I20240507 02:11:30.919535 22362 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20240507 02:11:30.943384 22238 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:11:30.941740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60990 (local address 127.31.131.125:45921)
0507 02:11:30.941786 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:30.941788 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:30.942386 (+ 598us) server_negotiation.cc:407] Connection header received
0507 02:11:30.942505 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:30.942507 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:30.942517 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:30.942825 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:30.943315 (+ 490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60990: BlockingRecv error: recv got EOF from 127.0.0.1:60990 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20240507 02:11:30.944162 22313 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:11:30.942086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33654 (local address 127.31.131.124:42983)
0507 02:11:30.942128 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:30.942130 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:30.943640 (+ 1510us) server_negotiation.cc:407] Connection header received
0507 02:11:30.943737 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:30.943738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:30.943749 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:30.943776 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:30.944108 (+ 332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33654: BlockingRecv error: recv got EOF from 127.0.0.1:33654 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20240507 02:11:30.947623 22162 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:11:30.946867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48932 (local address 127.31.131.126:39745)
0507 02:11:30.946909 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:30.946910 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:30.947079 (+ 169us) server_negotiation.cc:407] Connection header received
0507 02:11:30.947172 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:30.947173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:30.947184 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:30.947212 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:30.947565 (+ 353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48932: BlockingRecv error: recv got EOF from 127.0.0.1:48932 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20240507 02:11:30.955587 22238 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:11:30.954829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32772 (local address 127.31.131.125:45921)
0507 02:11:30.954871 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:30.954873 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:30.954990 (+ 117us) server_negotiation.cc:407] Connection header received
0507 02:11:30.955102 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:30.955103 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:30.955119 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:30.955149 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:30.955527 (+ 378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32772: BlockingRecv error: recv got EOF from 127.0.0.1:32772 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20240507 02:11:30.958650 22238 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:11:30.957868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32786 (local address 127.31.131.125:45921)
0507 02:11:30.957903 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:30.957905 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:30.958115 (+ 210us) server_negotiation.cc:407] Connection header received
0507 02:11:30.958210 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:30.958210 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:30.958220 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:30.958248 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:30.958598 (+ 350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32786: BlockingRecv error: recv got EOF from 127.0.0.1:32786 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20240507 02:11:30.962618 22238 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:11:30.961834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32796 (local address 127.31.131.125:45921)
0507 02:11:30.961866 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:30.961867 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:30.962070 (+ 203us) server_negotiation.cc:407] Connection header received
0507 02:11:30.962162 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:30.962163 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:30.962173 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:30.962199 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:30.962519 (+ 320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32796: BlockingRecv error: recv got EOF from 127.0.0.1:32796 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20240507 02:11:30.968523 22238 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:11:30.967799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32808 (local address 127.31.131.125:45921)
0507 02:11:30.967834 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:30.967835 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:30.968049 (+ 214us) server_negotiation.cc:407] Connection header received
0507 02:11:30.968138 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:30.968139 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:30.968149 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:30.968176 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:30.968474 (+ 298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32808: BlockingRecv error: recv got EOF from 127.0.0.1:32808 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20240507 02:11:30.978655 22238 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:11:30.977875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32822 (local address 127.31.131.125:45921)
0507 02:11:30.977915 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:30.977917 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:30.978052 (+ 135us) server_negotiation.cc:407] Connection header received
0507 02:11:30.978128 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:30.978128 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:30.978139 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:30.978211 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:30.978598 (+ 387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32822: BlockingRecv error: recv got EOF from 127.0.0.1:32822 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20240507 02:11:30.996486 22238 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:11:30.995756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32832 (local address 127.31.131.125:45921)
0507 02:11:30.995802 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:30.995804 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:30.995927 (+ 123us) server_negotiation.cc:407] Connection header received
0507 02:11:30.996013 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:30.996014 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:30.996026 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:30.996077 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:30.996414 (+ 337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32832: BlockingRecv error: recv got EOF from 127.0.0.1:32832 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20240507 02:11:31.030364 22238 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:11:31.029739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32840 (local address 127.31.131.125:45921)
0507 02:11:31.029800 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:31.029802 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:31.029870 (+ 68us) server_negotiation.cc:407] Connection header received
0507 02:11:31.029907 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:31.029908 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:31.029920 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:31.029990 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:31.030300 (+ 310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32840: BlockingRecv error: recv got EOF from 127.0.0.1:32840 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20240507 02:11:31.096031 22238 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:11:31.095450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32854 (local address 127.31.131.125:45921)
0507 02:11:31.095509 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:31.095511 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:31.095581 (+ 70us) server_negotiation.cc:407] Connection header received
0507 02:11:31.095628 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:31.095629 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:31.095639 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:31.095704 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:31.095979 (+ 275us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32854: BlockingRecv error: recv got EOF from 127.0.0.1:32854 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20240507 02:11:31.225838 22238 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:11:31.225199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32864 (local address 127.31.131.125:45921)
0507 02:11:31.225277 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:31.225280 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:31.225315 (+ 35us) server_negotiation.cc:407] Connection header received
0507 02:11:31.225396 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:31.225397 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:31.225408 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:31.225454 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:31.225789 (+ 335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32864: BlockingRecv error: recv got EOF from 127.0.0.1:32864 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20240507 02:11:31.483567 22238 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:11:31.483042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32876 (local address 127.31.131.125:45921)
0507 02:11:31.483100 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:31.483102 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:31.483115 (+ 13us) server_negotiation.cc:407] Connection header received
0507 02:11:31.483179 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:31.483180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:31.483189 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:31.483247 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:31.483518 (+ 271us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32876: BlockingRecv error: recv got EOF from 127.0.0.1:32876 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20240507 02:11:31.781108 22327 catalog_manager.cc:1259] Loaded cluster ID: cff42734dc304863824edc5cc36204cc
I20240507 02:11:31.781181 22327 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6346d184c7d84fcfafb99e03d9b29d13: loading cluster ID for follower catalog manager: success
I20240507 02:11:31.781989 22327 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6346d184c7d84fcfafb99e03d9b29d13: acquiring CA information for follower catalog manager: success
I20240507 02:11:31.782238 22327 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6346d184c7d84fcfafb99e03d9b29d13: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240507 02:11:31.796090 22341 catalog_manager.cc:1259] Loaded cluster ID: cff42734dc304863824edc5cc36204cc
I20240507 02:11:31.796156 22341 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ef114a74cfa34db29288c61ea16c85a6: loading cluster ID for follower catalog manager: success
I20240507 02:11:31.797004 22341 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ef114a74cfa34db29288c61ea16c85a6: acquiring CA information for follower catalog manager: success
I20240507 02:11:31.797262 22341 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ef114a74cfa34db29288c61ea16c85a6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240507 02:11:31.998824 22438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:31.996735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39642 (local address 127.31.131.125:45921)
0507 02:11:31.996875 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:31.996877 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:31.996887 (+ 10us) server_negotiation.cc:407] Connection header received
0507 02:11:31.996916 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:31.996916 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:31.996956 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:31.997030 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0507 02:11:31.997282 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:31.997776 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:31.998320 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:31.998435 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:31.998473 (+ 38us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0507 02:11:31.998593 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0507 02:11:31.998652 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:31.998717 (+ 65us) server_negotiation.cc:299] Negotiation successful
0507 02:11:31.998739 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":67,"threads_started":1}
I20240507 02:11:32.002379 22444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:32.000539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33296 (local address 127.31.131.124:42983)
0507 02:11:32.000657 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:32.000659 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:32.000666 (+ 7us) server_negotiation.cc:407] Connection header received
0507 02:11:32.000705 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:32.000706 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:32.000730 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:32.000799 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0507 02:11:32.000992 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:32.001488 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:32.001980 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:32.002104 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:32.002139 (+ 35us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0507 02:11:32.002222 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0507 02:11:32.002282 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:32.002298 (+ 16us) server_negotiation.cc:299] Negotiation successful
0507 02:11:32.002318 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":62,"threads_started":1}
I20240507 02:11:32.006062 22438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:32.003823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39644 (local address 127.31.131.125:45921)
0507 02:11:32.003875 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:32.003876 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:32.004014 (+ 138us) server_negotiation.cc:407] Connection header received
0507 02:11:32.004063 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:32.004064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:32.004078 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:32.004120 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0507 02:11:32.004426 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:32.004955 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:32.005636 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:32.005801 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:32.005821 (+ 20us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0507 02:11:32.005847 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0507 02:11:32.005890 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:32.005977 (+ 87us) server_negotiation.cc:299] Negotiation successful
0507 02:11:32.006000 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20240507 02:11:32.006274 22444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:32.004108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33310 (local address 127.31.131.124:42983)
0507 02:11:32.004144 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:32.004146 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:32.004241 (+ 95us) server_negotiation.cc:407] Connection header received
0507 02:11:32.004280 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:32.004281 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:32.004291 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:32.004339 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0507 02:11:32.004632 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:32.005256 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:32.005849 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:32.006006 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:32.006024 (+ 18us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0507 02:11:32.006046 (+ 22us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0507 02:11:32.006104 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:32.006186 (+ 82us) server_negotiation.cc:299] Negotiation successful
0507 02:11:32.006211 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20240507 02:11:32.006490 22450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:32.003816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53396 (local address 127.31.131.126:39745)
0507 02:11:32.003912 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:32.003914 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:32.003923 (+ 9us) server_negotiation.cc:407] Connection header received
0507 02:11:32.003969 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:32.003970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:32.003991 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:32.004066 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0507 02:11:32.004386 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:32.005073 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:32.005728 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:32.005908 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:32.005965 (+ 57us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0507 02:11:32.006056 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0507 02:11:32.006113 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:32.006384 (+ 271us) server_negotiation.cc:299] Negotiation successful
0507 02:11:32.006418 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79,"thread_start_us":49,"threads_started":1}
I20240507 02:11:32.007907 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 22091
I20240507 02:11:32.014361 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 22166
I20240507 02:11:32.018733 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 22241
2024-05-07T02:11:32Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2622 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715047604443657-32269-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 07 02:11:32 dist-test-slave-v0bg krb5kdc[22455](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 07 02:11:32 dist-test-slave-v0bg krb5kdc[22455](info): set up 2 sockets
May 07 02:11:32 dist-test-slave-v0bg krb5kdc[22455](info): commencing operation
krb5kdc: starting...
W20240507 02:11:32.907456 32269 mini_kdc.cc:121] Time spent starting KDC: real 0.870s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715047604443657-32269-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715047604443657-32269-0/krb5kdc/test-user.keytab.
May 07 02:11:32 dist-test-slave-v0bg krb5kdc[22455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2024-05-07T02:11:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-07T02:11:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.131.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab.
Entry for principal kudu/127.31.131.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab.
I20240507 02:11:33.007433 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715047604443657-32269-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715047604443657-32269-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715047604443657-32269-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:34013
--webserver_interface=127.31.131.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab
--principal=kudu/127.31.131.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:37651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.131.126:34013,127.31.131.125:38391,127.31.131.124:44023
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20240507 02:11:33.082444 22468 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20240507 02:11:33.082676 22468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:11:33.082703 22468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:11:33.082732 22468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:11:33.084046 22468 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240507 02:11:33.084090 22468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:11:33.084108 22468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:11:33.084124 22468 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240507 02:11:33.084137 22468 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240507 02:11:33.085599 22468 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:37651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715047604443657-32269-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715047604443657-32269-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.131.126:34013,127.31.131.125:38391,127.31.131.124:44023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.126.keytab
--principal=kudu/127.31.131.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.126:34013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.131.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715047604443657-32269-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:11:33.085804 22468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:11:33.085976 22468 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240507 02:11:33.089931 22468 server_base.cc:1034] running on GCE node
W20240507 02:11:33.089865 22474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:11:33.089869 22476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:11:33.090129 22473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:33.090340 22468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:11:33.090584 22468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:11:33.091706 22468 hybrid_clock.cc:648] HybridClock initialized: now 1715047893091698 us; error 27 us; skew 500 ppm
May 07 02:11:33 dist-test-slave-v0bg krb5kdc[22455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047893, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:11:33.093891 22468 init.cc:377] Logged in from keytab as kudu/127.31.131.126@KRBTEST.COM (short username kudu)
I20240507 02:11:33.094311 22468 webserver.cc:460] Webserver started at http://127.31.131.126:43649/ using document root <none> and password file <none>
I20240507 02:11:33.094482 22468 fs_manager.cc:359] Metadata directory not provided
I20240507 02:11:33.094523 22468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:11:33.094658 22468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:11:33.095624 22468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715047604443657-32269-0/minicluster-data/master-0/data/instance:
uuid: "f621d02b2c7d4f1799d6549aa5c9a6e0"
format_stamp: "Formatted at 2024-05-07 02:11:33 on dist-test-slave-v0bg"
server_key: "4b8fd2289958955910e09b6f158f56c2"
server_key_iv: "0102ed62b9dc5eafd3510d0b17b4249b"
server_key_version: "encryptionkey@0"
I20240507 02:11:33.095907 22468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715047604443657-32269-0/minicluster-data/master-0/wal/instance:
uuid: "f621d02b2c7d4f1799d6549aa5c9a6e0"
format_stamp: "Formatted at 2024-05-07 02:11:33 on dist-test-slave-v0bg"
server_key: "4b8fd2289958955910e09b6f158f56c2"
server_key_iv: "0102ed62b9dc5eafd3510d0b17b4249b"
server_key_version: "encryptionkey@0"
I20240507 02:11:33.096968 22468 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20240507 02:11:33.097600 22483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:33.097754 22468 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:33.097823 22468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715047604443657-32269-0/minicluster-data/master-0/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715047604443657-32269-0/minicluster-data/master-0/wal
uuid: "f621d02b2c7d4f1799d6549aa5c9a6e0"
format_stamp: "Formatted at 2024-05-07 02:11:33 on dist-test-slave-v0bg"
server_key: "4b8fd2289958955910e09b6f158f56c2"
server_key_iv: "0102ed62b9dc5eafd3510d0b17b4249b"
server_key_version: "encryptionkey@0"
I20240507 02:11:33.097878 22468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715047604443657-32269-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715047604443657-32269-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715047604443657-32269-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:11:33.109072 22468 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:11:33.110742 22468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:11:33.110855 22468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:11:33.114167 22468 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.126:34013
I20240507 02:11:33.114197 22535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.126:34013 every 8 connection(s)
I20240507 02:11:33.114499 22468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715047604443657-32269-0/minicluster-data/master-0/data/info.pb
I20240507 02:11:33.115335 22536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:11:33.116710 22536 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.131.126" port: 34013 } has no permanent_uuid. Determining permanent_uuid...
I20240507 02:11:33.120810 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 22468
I20240507 02:11:33.120909 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715047604443657-32269-0/minicluster-data/master-0/wal/instance
I20240507 02:11:33.121073 32269 external_mini_cluster.cc:1428] Setting key 61a5f802b372bf733acab1453fa57ce8
May 07 02:11:33 dist-test-slave-v0bg krb5kdc[22455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047893, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.126@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:11:33.124963 22538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:33.117281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:34013 (local address 127.0.0.1:46338)
0507 02:11:33.117446 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:33.117470 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0507 02:11:33.117896 (+ 426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:11:33.118114 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:11:33.118116 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:11:33.118417 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:11:33.118690 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:33.118695 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:33.119465 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:11:33.119465 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:11:33.119822 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:33.119824 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:33.119923 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:33.120372 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:11:33.120385 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:11:33.121836 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:11:33.123779 (+ 1943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:11:33.123785 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:11:33.123787 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:11:33.123953 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:11:33.124158 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:11:33.124158 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:11:33.124159 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:11:33.124206 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:11:33.124567 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:11:33.124571 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:11:33.124626 (+ 55us) client_negotiation.cc:769] Sending connection context
0507 02:11:33.124702 (+ 76us) client_negotiation.cc:240] Negotiation successful
0507 02:11:33.124815 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":64,"threads_started":1}
I20240507 02:11:33.125030 22539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:33.117321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46338 (local address 127.31.131.126:34013)
0507 02:11:33.117456 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:33.117466 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:33.117539 (+ 73us) server_negotiation.cc:407] Connection header received
0507 02:11:33.118004 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:33.118005 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:33.118021 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:33.118101 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:33.118765 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:33.119399 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:33.119880 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:33.120023 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:33.121925 (+ 1902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:33.121942 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:33.121943 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:11:33.121949 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:33.123698 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:33.124031 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:33.124034 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:33.124034 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:33.124092 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:33.124258 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:33.124259 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:33.124259 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:33.124504 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:33.124592 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:33.124748 (+ 156us) server_negotiation.cc:299] Negotiation successful
0507 02:11:33.124891 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":106,"thread_start_us":70,"threads_started":1}
I20240507 02:11:33.125999 22536 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.131.125" port: 38391 } has no permanent_uuid. Determining permanent_uuid...
I20240507 02:11:33.126338 22538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:33.126232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.125:38391 (local address 127.0.0.1:41380)
0507 02:11:33.126275 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:33.126314 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.125:38391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20240507 02:11:33.126451 22486 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.131.125:38391: connect: Connection refused (error 111)
I20240507 02:11:33.126868 22538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:33.126785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.125:38391 (local address 127.0.0.1:41390)
0507 02:11:33.126826 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:33.126850 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.125:38391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20240507 02:11:33.126965 22536 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.31.131.125:38391: Network error: Client connection negotiation failed: client connection to 127.31.131.125:38391: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.31.131.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.125.keytab.
Entry for principal kudu/127.31.131.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.125.keytab.
I20240507 02:11:33.143309 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715047604443657-32269-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715047604443657-32269-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715047604443657-32269-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715047604443657-32269-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.125:38391
--webserver_interface=127.31.131.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.125.keytab
--principal=kudu/127.31.131.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:37651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.131.126:34013,127.31.131.125:38391,127.31.131.124:44023
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240507 02:11:33.176182 22536 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.131.125" port: 38391 } attempt: 1
I20240507 02:11:33.176689 22538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:33.176561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.125:38391 (local address 127.0.0.1:41396)
0507 02:11:33.176622 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:33.176655 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.125:38391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20240507 02:11:33.177073 22538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:33.177000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.125:38391 (local address 127.0.0.1:41408)
0507 02:11:33.177034 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:33.177057 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.125:38391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20240507 02:11:33.177153 22536 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.31.131.125:38391: Network error: Client connection negotiation failed: client connection to 127.31.131.125:38391: connect: Connection refused (error 111)
W20240507 02:11:33.219678 22543 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20240507 02:11:33.219863 22543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:11:33.219877 22543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:11:33.219895 22543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:11:33.221127 22543 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240507 02:11:33.221161 22543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:11:33.221172 22543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:11:33.221184 22543 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240507 02:11:33.221192 22543 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240507 02:11:33.222640 22543 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:37651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715047604443657-32269-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715047604443657-32269-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.31.131.126:34013,127.31.131.125:38391,127.31.131.124:44023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.125.keytab
--principal=kudu/127.31.131.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.125:38391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715047604443657-32269-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.31.131.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715047604443657-32269-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:11:33.222812 22543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:11:33.222954 22543 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:11:33.226563 22549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:11:33.226591 22548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:33.226701 22543 server_base.cc:1034] running on GCE node
W20240507 02:11:33.226816 22551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:33.227026 22543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:11:33.227257 22543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:11:33.228389 22543 hybrid_clock.cc:648] HybridClock initialized: now 1715047893228368 us; error 36 us; skew 500 ppm
May 07 02:11:33 dist-test-slave-v0bg krb5kdc[22455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047893, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:11:33.234225 22543 init.cc:377] Logged in from keytab as kudu/127.31.131.125@KRBTEST.COM (short username kudu)
I20240507 02:11:33.234710 22543 webserver.cc:460] Webserver started at http://127.31.131.125:35383/ using document root <none> and password file <none>
I20240507 02:11:33.234885 22543 fs_manager.cc:359] Metadata directory not provided
I20240507 02:11:33.234926 22543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:11:33.235033 22543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:11:33.235991 22543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715047604443657-32269-0/minicluster-data/master-1/data/instance:
uuid: "c4d97049ef4f470bab82a54a275f100b"
format_stamp: "Formatted at 2024-05-07 02:11:33 on dist-test-slave-v0bg"
server_key: "83fe4f1dfeb830d1aa9fe095cdba6976"
server_key_iv: "3945067d3a7635cdc23c61d36566fa3d"
server_key_version: "encryptionkey@0"
I20240507 02:11:33.236266 22543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715047604443657-32269-0/minicluster-data/master-1/wal/instance:
uuid: "c4d97049ef4f470bab82a54a275f100b"
format_stamp: "Formatted at 2024-05-07 02:11:33 on dist-test-slave-v0bg"
server_key: "83fe4f1dfeb830d1aa9fe095cdba6976"
server_key_iv: "3945067d3a7635cdc23c61d36566fa3d"
server_key_version: "encryptionkey@0"
I20240507 02:11:33.237396 22543 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20240507 02:11:33.238021 22558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:33.238174 22543 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240507 02:11:33.238247 22543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715047604443657-32269-0/minicluster-data/master-1/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715047604443657-32269-0/minicluster-data/master-1/wal
uuid: "c4d97049ef4f470bab82a54a275f100b"
format_stamp: "Formatted at 2024-05-07 02:11:33 on dist-test-slave-v0bg"
server_key: "83fe4f1dfeb830d1aa9fe095cdba6976"
server_key_iv: "3945067d3a7635cdc23c61d36566fa3d"
server_key_version: "encryptionkey@0"
I20240507 02:11:33.238306 22543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715047604443657-32269-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715047604443657-32269-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715047604443657-32269-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:11:33.245306 22536 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.131.125" port: 38391 } attempt: 2
I20240507 02:11:33.245715 22538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:33.245577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.125:38391 (local address 127.0.0.1:41420)
0507 02:11:33.245633 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:33.245677 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.125:38391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20240507 02:11:33.246012 22538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:33.245948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.125:38391 (local address 127.0.0.1:41434)
0507 02:11:33.245977 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:33.245997 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.125:38391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20240507 02:11:33.246090 22536 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.31.131.125:38391: Network error: Client connection negotiation failed: client connection to 127.31.131.125:38391: connect: Connection refused (error 111)
I20240507 02:11:33.248858 22543 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:11:33.250298 22543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:11:33.250397 22543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:11:33.253629 22543 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.125:38391
I20240507 02:11:33.253661 22610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.125:38391 every 8 connection(s)
I20240507 02:11:33.253965 22543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715047604443657-32269-0/minicluster-data/master-1/data/info.pb
I20240507 02:11:33.254773 22611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:11:33.256132 22611 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.131.126" port: 34013 } has no permanent_uuid. Determining permanent_uuid...
I20240507 02:11:33.257311 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 22543
I20240507 02:11:33.257403 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715047604443657-32269-0/minicluster-data/master-1/wal/instance
I20240507 02:11:33.257565 32269 external_mini_cluster.cc:1428] Setting key a9d46537d4921afb80b5cabfe790435c
May 07 02:11:33 dist-test-slave-v0bg krb5kdc[22455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047893, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.125@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.131.124@KRBTEST.COM; defaulting to no policy
I20240507 02:11:33.264292 22539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:33.256742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46354 (local address 127.31.131.126:34013)
0507 02:11:33.256801 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:33.256804 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:33.256955 (+ 151us) server_negotiation.cc:407] Connection header received
0507 02:11:33.257433 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:33.257435 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:33.257446 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:33.257520 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:33.258287 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:33.258903 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:33.259560 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:33.259676 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:33.261807 (+ 2131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:33.261812 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:33.261812 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:11:33.261818 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:33.263097 (+ 1279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:33.263510 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:33.263511 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:33.263511 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:33.263528 (+ 17us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:33.263717 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:33.263717 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:33.263717 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:33.263800 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:33.263841 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:33.264136 (+ 295us) server_negotiation.cc:299] Negotiation successful
0507 02:11:33.264203 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20240507 02:11:33.264458 22613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:33.256687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:34013 (local address 127.0.0.1:46354)
0507 02:11:33.256849 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:33.256876 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0507 02:11:33.257316 (+ 440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:11:33.257584 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:11:33.257588 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:11:33.257915 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:11:33.258190 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:33.258194 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:33.258985 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:11:33.258986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:11:33.259486 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:33.259488 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:33.259629 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:33.260130 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:11:33.260152 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:11:33.261701 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:11:33.263174 (+ 1473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:11:33.263181 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:11:33.263184 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:11:33.263442 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:11:33.263580 (+ 138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:11:33.263581 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:11:33.263581 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:11:33.263661 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:11:33.263852 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:11:33.263857 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:11:33.264012 (+ 155us) client_negotiation.cc:769] Sending connection context
0507 02:11:33.264130 (+ 118us) client_negotiation.cc:240] Negotiation successful
0507 02:11:33.264266 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":80,"threads_started":1}
I20240507 02:11:33.265017 22611 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.131.125" port: 38391 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.131.124@KRBTEST.COM" created.
May 07 02:11:33 dist-test-slave-v0bg krb5kdc[22455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047893, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.125@KRBTEST.COM for kudu/127.31.131.125@KRBTEST.COM
I20240507 02:11:33.271528 22613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:33.265266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.125:38391 (local address 127.0.0.1:41450)
0507 02:11:33.265299 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:33.265310 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0507 02:11:33.265348 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:11:33.265592 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:11:33.265594 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:11:33.265717 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:11:33.265857 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:33.265859 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:33.266807 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:11:33.266809 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:11:33.267235 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:33.267238 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:33.267306 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:33.267727 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:11:33.267732 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:11:33.269040 (+ 1308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:11:33.270720 (+ 1680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:11:33.270720 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:11:33.270721 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:11:33.270846 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:11:33.270993 (+ 147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:11:33.270993 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:11:33.270994 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:11:33.271020 (+ 26us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:11:33.271222 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:11:33.271223 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:11:33.271256 (+ 33us) client_negotiation.cc:769] Sending connection context
0507 02:11:33.271320 (+ 64us) client_negotiation.cc:240] Negotiation successful
0507 02:11:33.271427 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16}
I20240507 02:11:33.271606 22615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:33.265332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41450 (local address 127.31.131.125:38391)
0507 02:11:33.265485 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:33.265490 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:33.265501 (+ 11us) server_negotiation.cc:407] Connection header received
0507 02:11:33.265517 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:33.265518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:33.265529 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:33.265585 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:33.265921 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:33.266743 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:33.267328 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:33.267531 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:33.269123 (+ 1592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:33.269131 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:33.269131 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:11:33.269136 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:33.270659 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:33.270912 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:33.270915 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:33.270915 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:33.270945 (+ 30us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:33.271067 (+ 122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:33.271068 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:33.271068 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:33.271172 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:33.271245 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:33.271349 (+ 104us) server_negotiation.cc:299] Negotiation successful
0507 02:11:33.271499 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":129,"thread_start_us":86,"threads_started":1}
I20240507 02:11:33.272078 22611 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.131.124" port: 44023 } has no permanent_uuid. Determining permanent_uuid...
I20240507 02:11:33.272400 22613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:33.272271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.124:44023 (local address 127.0.0.1:53170)
0507 02:11:33.272321 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:33.272370 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.124:44023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20240507 02:11:33.272523 22559 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.131.124:44023: connect: Connection refused (error 111)
I20240507 02:11:33.272893 22613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:33.272805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.124:44023 (local address 127.0.0.1:53180)
0507 02:11:33.272844 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:33.272870 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.124:44023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20240507 02:11:33.272984 22611 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.31.131.124:44023: Network error: Client connection negotiation failed: client connection to 127.31.131.124:44023: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.131.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.124.keytab.
Entry for principal kudu/127.31.131.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.124.keytab.
I20240507 02:11:33.279042 32269 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTztKno/build/release/bin/kudu
/tmp/dist-test-taskTztKno/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715047604443657-32269-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715047604443657-32269-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715047604443657-32269-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715047604443657-32269-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.124:44023
--webserver_interface=127.31.131.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.124.keytab
--principal=kudu/127.31.131.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.131.84:37651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.131.126:34013,127.31.131.125:38391,127.31.131.124:44023
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715047604443657-32269-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715047604443657-32269-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715047604443657-32269-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20240507 02:11:33.322201 22611 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.131.124" port: 44023 } attempt: 1
I20240507 02:11:33.322752 22613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:33.322602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.124:44023 (local address 127.0.0.1:53196)
0507 02:11:33.322666 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:33.322715 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.124:44023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20240507 02:11:33.323057 22613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:33.322977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.124:44023 (local address 127.0.0.1:53210)
0507 02:11:33.323013 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:33.323039 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.124:44023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20240507 02:11:33.323159 22611 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.31.131.124:44023: Network error: Client connection negotiation failed: client connection to 127.31.131.124:44023: connect: Connection refused (error 111)
I20240507 02:11:33.337240 22536 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.131.125" port: 38391 } attempt: 3
May 07 02:11:33 dist-test-slave-v0bg krb5kdc[22455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047893, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.126@KRBTEST.COM for kudu/127.31.131.125@KRBTEST.COM
I20240507 02:11:33.343796 22538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:33.337602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.125:38391 (local address 127.0.0.1:41460)
0507 02:11:33.337666 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:33.337679 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0507 02:11:33.337730 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:11:33.337896 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:11:33.337897 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:11:33.338051 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:11:33.338219 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:33.338221 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:33.339087 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:11:33.339087 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:11:33.339439 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:33.339440 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:33.339495 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:33.339937 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:11:33.339941 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:11:33.341535 (+ 1594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:11:33.343026 (+ 1491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:11:33.343027 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:11:33.343028 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:11:33.343150 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:11:33.343281 (+ 131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:11:33.343281 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:11:33.343282 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:11:33.343310 (+ 28us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:11:33.343496 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:11:33.343497 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:11:33.343541 (+ 44us) client_negotiation.cc:769] Sending connection context
0507 02:11:33.343598 (+ 57us) client_negotiation.cc:240] Negotiation successful
0507 02:11:33.343697 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20240507 02:11:33.343796 22615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:33.337693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41460 (local address 127.31.131.125:38391)
0507 02:11:33.337749 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:33.337751 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:33.337762 (+ 11us) server_negotiation.cc:407] Connection header received
0507 02:11:33.337792 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:33.337793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:33.337808 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:33.337879 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:33.338297 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:33.339007 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:33.339508 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:33.339682 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:33.341611 (+ 1929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:33.341617 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:33.341617 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:11:33.341625 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:33.342953 (+ 1328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:33.343210 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:33.343211 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:33.343212 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:33.343233 (+ 21us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:33.343355 (+ 122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:33.343356 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:33.343356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:33.343447 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:33.343500 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:33.343625 (+ 125us) server_negotiation.cc:299] Negotiation successful
0507 02:11:33.343697 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20240507 02:11:33.344784 22536 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.131.124" port: 44023 } has no permanent_uuid. Determining permanent_uuid...
I20240507 02:11:33.345125 22538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:33.345036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.124:44023 (local address 127.0.0.1:53226)
0507 02:11:33.345077 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:33.345102 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.124:44023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20240507 02:11:33.345475 22538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:33.345364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.124:44023 (local address 127.0.0.1:53236)
0507 02:11:33.345429 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:33.345454 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.124:44023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20240507 02:11:33.345559 22536 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.31.131.124:44023: Network error: Client connection negotiation failed: client connection to 127.31.131.124:44023: connect: Connection refused (error 111)
W20240507 02:11:33.357407 22618 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20240507 02:11:33.357610 22618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 02:11:33.357625 22618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 02:11:33.357643 22618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 02:11:33.358973 22618 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240507 02:11:33.359019 22618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20240507 02:11:33.359030 22618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 02:11:33.359041 22618 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240507 02:11:33.359059 22618 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240507 02:11:33.360548 22618 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.131.84:37651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715047604443657-32269-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715047604443657-32269-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.31.131.126:34013,127.31.131.125:38391,127.31.131.124:44023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715047604443657-32269-0/krb5kdc/kudu_127.31.131.124.keytab
--principal=kudu/127.31.131.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.131.124:44023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715047604443657-32269-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.31.131.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22618
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715047604443657-32269-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type RELEASE
built by None at 07 May 2024 01:43:16 UTC on efd1337a2aa3
build id 170
I20240507 02:11:33.360723 22618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 02:11:33.360872 22618 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 02:11:33.364650 22626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 02:11:33.364650 22624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:33.364735 22618 server_base.cc:1034] running on GCE node
W20240507 02:11:33.364845 22623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 02:11:33.365152 22618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 02:11:33.365379 22618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 02:11:33.366514 22618 hybrid_clock.cc:648] HybridClock initialized: now 1715047893366503 us; error 30 us; skew 500 ppm
May 07 02:11:33 dist-test-slave-v0bg krb5kdc[22455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047893, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20240507 02:11:33.368736 22618 init.cc:377] Logged in from keytab as kudu/127.31.131.124@KRBTEST.COM (short username kudu)
I20240507 02:11:33.369122 22618 webserver.cc:460] Webserver started at http://127.31.131.124:35807/ using document root <none> and password file <none>
I20240507 02:11:33.369277 22618 fs_manager.cc:359] Metadata directory not provided
I20240507 02:11:33.369315 22618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 02:11:33.369415 22618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 02:11:33.370456 22618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715047604443657-32269-0/minicluster-data/master-2/data/instance:
uuid: "7615f7659f344c2aaafd8c723339b4e8"
format_stamp: "Formatted at 2024-05-07 02:11:33 on dist-test-slave-v0bg"
server_key: "fdf1a5e56739e6d857a55d72742efc6b"
server_key_iv: "bc08290ea61c4f6b0cd79d22d3b42cf3"
server_key_version: "encryptionkey@0"
I20240507 02:11:33.370841 22618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715047604443657-32269-0/minicluster-data/master-2/wal/instance:
uuid: "7615f7659f344c2aaafd8c723339b4e8"
format_stamp: "Formatted at 2024-05-07 02:11:33 on dist-test-slave-v0bg"
server_key: "fdf1a5e56739e6d857a55d72742efc6b"
server_key_iv: "bc08290ea61c4f6b0cd79d22d3b42cf3"
server_key_version: "encryptionkey@0"
I20240507 02:11:33.371920 22618 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240507 02:11:33.372555 22633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 02:11:33.372699 22618 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240507 02:11:33.372774 22618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715047604443657-32269-0/minicluster-data/master-2/data,/tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715047604443657-32269-0/minicluster-data/master-2/wal
uuid: "7615f7659f344c2aaafd8c723339b4e8"
format_stamp: "Formatted at 2024-05-07 02:11:33 on dist-test-slave-v0bg"
server_key: "fdf1a5e56739e6d857a55d72742efc6b"
server_key_iv: "bc08290ea61c4f6b0cd79d22d3b42cf3"
server_key_version: "encryptionkey@0"
I20240507 02:11:33.372833 22618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715047604443657-32269-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715047604443657-32269-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715047604443657-32269-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 02:11:33.376679 22536 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.131.124" port: 44023 } attempt: 1
I20240507 02:11:33.377076 22538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:33.376973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.124:44023 (local address 127.0.0.1:53248)
0507 02:11:33.377020 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:33.377048 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.124:44023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20240507 02:11:33.377357 22538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:33.377287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.124:44023 (local address 127.0.0.1:53258)
0507 02:11:33.377318 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:33.377339 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.124:44023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20240507 02:11:33.377444 22536 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.31.131.124:44023: Network error: Client connection negotiation failed: client connection to 127.31.131.124:44023: connect: Connection refused (error 111)
I20240507 02:11:33.387487 22618 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 02:11:33.389114 22618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 02:11:33.389228 22618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 02:11:33.391315 22611 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.131.124" port: 44023 } attempt: 2
I20240507 02:11:33.391738 22613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:33.391608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.124:44023 (local address 127.0.0.1:53268)
0507 02:11:33.391667 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:33.391706 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.124:44023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20240507 02:11:33.392084 22613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:33.391993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.124:44023 (local address 127.0.0.1:53276)
0507 02:11:33.392037 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:33.392066 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.131.124:44023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20240507 02:11:33.392171 22611 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.31.131.124:44023: Network error: Client connection negotiation failed: client connection to 127.31.131.124:44023: connect: Connection refused (error 111)
I20240507 02:11:33.392690 22618 rpc_server.cc:307] RPC server started. Bound to: 127.31.131.124:44023
I20240507 02:11:33.392717 22685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.131.124:44023 every 8 connection(s)
I20240507 02:11:33.393018 22618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715047604443657-32269-0/minicluster-data/master-2/data/info.pb
I20240507 02:11:33.393772 22686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 02:11:33.395190 22686 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.131.126" port: 34013 } has no permanent_uuid. Determining permanent_uuid...
May 07 02:11:33 dist-test-slave-v0bg krb5kdc[22455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047893, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.124@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:11:33.402387 22539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:33.395797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46360 (local address 127.31.131.126:34013)
0507 02:11:33.395855 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:33.395857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:33.396002 (+ 145us) server_negotiation.cc:407] Connection header received
0507 02:11:33.396481 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:33.396482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:33.396495 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:33.396545 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:33.397254 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:33.397684 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:33.398173 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:33.398326 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:33.400140 (+ 1814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:33.400143 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:33.400144 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:11:33.400149 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:33.401323 (+ 1174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:33.401663 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:33.401664 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:33.401664 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:33.401685 (+ 21us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:33.401835 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:33.401836 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:33.401837 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:33.401927 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:33.401984 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:33.402212 (+ 228us) server_negotiation.cc:299] Negotiation successful
0507 02:11:33.402292 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20240507 02:11:33.402487 22688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:33.395758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.126:34013 (local address 127.0.0.1:46360)
0507 02:11:33.395903 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:33.395930 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0507 02:11:33.396371 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:11:33.396604 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:11:33.396608 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:11:33.396896 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:11:33.397165 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:33.397169 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:33.397756 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:11:33.397757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:11:33.398104 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:33.398105 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:33.398230 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:33.398761 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:11:33.398778 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:11:33.400056 (+ 1278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:11:33.401396 (+ 1340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:11:33.401402 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:11:33.401405 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:11:33.401600 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:11:33.401734 (+ 134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:11:33.401735 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:11:33.401735 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:11:33.401788 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:11:33.401981 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:11:33.401986 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:11:33.402097 (+ 111us) client_negotiation.cc:769] Sending connection context
0507 02:11:33.402188 (+ 91us) client_negotiation.cc:240] Negotiation successful
0507 02:11:33.402304 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":69,"threads_started":1}
I20240507 02:11:33.402961 32269 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTztKno/build/release/bin/kudu as pid 22618
I20240507 02:11:33.403050 32269 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTztKno/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1715047604443657-32269-0/minicluster-data/master-2/wal/instance
I20240507 02:11:33.403016 22686 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.131.125" port: 38391 } has no permanent_uuid. Determining permanent_uuid...
I20240507 02:11:33.403218 32269 external_mini_cluster.cc:1428] Setting key d7db8fcf4d13ccf27d8f77585e04d641
May 07 02:11:33 dist-test-slave-v0bg krb5kdc[22455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047893, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.124@KRBTEST.COM for kudu/127.31.131.125@KRBTEST.COM
May 07 02:11:33 dist-test-slave-v0bg krb5kdc[22455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.131.124@KRBTEST.COM
I20240507 02:11:33.408658 22688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:33.403258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.125:38391 (local address 127.0.0.1:41474)
0507 02:11:33.403302 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:33.403311 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0507 02:11:33.403344 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:11:33.403530 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:11:33.403531 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:11:33.403662 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:11:33.403803 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:33.403804 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:33.404450 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:11:33.404450 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:11:33.404865 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:33.404867 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:33.404916 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:33.405288 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:11:33.405291 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:11:33.406450 (+ 1159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:11:33.407855 (+ 1405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:11:33.407856 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:11:33.407857 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:11:33.408068 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:11:33.408193 (+ 125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:11:33.408194 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:11:33.408195 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:11:33.408217 (+ 22us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:11:33.408386 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:11:33.408387 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:11:33.408428 (+ 41us) client_negotiation.cc:769] Sending connection context
0507 02:11:33.408479 (+ 51us) client_negotiation.cc:240] Negotiation successful
0507 02:11:33.408563 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20240507 02:11:33.408658 22615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:33.403320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41474 (local address 127.31.131.125:38391)
0507 02:11:33.403364 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:33.403366 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:33.403375 (+ 9us) server_negotiation.cc:407] Connection header received
0507 02:11:33.403400 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:33.403401 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:33.403412 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:33.403479 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:33.403878 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:33.404368 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:33.404927 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:33.405093 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:33.406522 (+ 1429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:33.406526 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:33.406526 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:11:33.406532 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:33.407788 (+ 1256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:33.408129 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:33.408130 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:33.408130 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:33.408151 (+ 21us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:33.408259 (+ 108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:33.408260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:33.408260 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:33.408341 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:33.408398 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:33.408502 (+ 104us) server_negotiation.cc:299] Negotiation successful
0507 02:11:33.408569 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20240507 02:11:33.409340 22686 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.131.124" port: 44023 } has no permanent_uuid. Determining permanent_uuid...
I20240507 02:11:33.410501 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:33.403541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53290 (local address 127.31.131.124:44023)
0507 02:11:33.403834 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:33.403842 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:33.403857 (+ 15us) server_negotiation.cc:407] Connection header received
0507 02:11:33.403875 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:33.403877 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:33.403900 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:33.403961 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:33.404623 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:33.405348 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:33.405967 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:33.406165 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:33.407913 (+ 1748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:33.407920 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:33.407921 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:11:33.407926 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:33.409402 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:33.409719 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:33.409722 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:33.409723 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:33.409757 (+ 34us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:33.409886 (+ 129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:33.409887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:33.409888 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:33.409990 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:33.410109 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:33.410195 (+ 86us) server_negotiation.cc:299] Negotiation successful
0507 02:11:33.410268 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":89,"threads_started":1}
May 07 02:11:33 dist-test-slave-v0bg krb5kdc[22455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047893, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.124@KRBTEST.COM for kudu/127.31.131.124@KRBTEST.COM
May 07 02:11:33 dist-test-slave-v0bg krb5kdc[22455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.131.126@KRBTEST.COM
I20240507 02:11:33.414716 22688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:33.409547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.124:44023 (local address 127.0.0.1:53300)
0507 02:11:33.409584 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:33.409593 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0507 02:11:33.409625 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:11:33.409854 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:11:33.409855 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:11:33.409972 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:11:33.410076 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:33.410077 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:33.410766 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:11:33.410767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:11:33.411099 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:33.411100 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:33.411154 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:33.411517 (+ 363us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:11:33.411520 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:11:33.412470 (+ 950us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:11:33.413980 (+ 1510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:11:33.413981 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:11:33.413981 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:11:33.414096 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:11:33.414248 (+ 152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:11:33.414249 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:11:33.414249 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:11:33.414267 (+ 18us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:11:33.414434 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:11:33.414435 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:11:33.414468 (+ 33us) client_negotiation.cc:769] Sending connection context
0507 02:11:33.414543 (+ 75us) client_negotiation.cc:240] Negotiation successful
0507 02:11:33.414634 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20240507 02:11:33.414876 22691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:33.409600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53300 (local address 127.31.131.124:44023)
0507 02:11:33.409747 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:33.409749 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:33.409757 (+ 8us) server_negotiation.cc:407] Connection header received
0507 02:11:33.409770 (+ 13us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:33.409770 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:33.409779 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:33.409850 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:33.410134 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:33.410678 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:33.411156 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:33.411311 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:33.412537 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:33.412540 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:33.412540 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:11:33.412545 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:33.413898 (+ 1353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:33.414168 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:33.414169 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:33.414170 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:33.414193 (+ 23us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:33.414298 (+ 105us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:33.414299 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:33.414299 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:33.414387 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:33.414451 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:33.414614 (+ 163us) server_negotiation.cc:299] Negotiation successful
0507 02:11:33.414719 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":100,"threads_started":1}
I20240507 02:11:33.416476 22686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7615f7659f344c2aaafd8c723339b4e8: Bootstrap starting.
I20240507 02:11:33.416971 22686 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7615f7659f344c2aaafd8c723339b4e8: Neither blocks nor log segments found. Creating new log.
I20240507 02:11:33.417186 22686 log.cc:824] T 00000000000000000000000000000000 P 7615f7659f344c2aaafd8c723339b4e8: Log is configured to *not* fsync() on all Append() calls
I20240507 02:11:33.417804 22686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7615f7659f344c2aaafd8c723339b4e8: No bootstrap required, opened a new log
I20240507 02:11:33.418673 22539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:33.411218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46372 (local address 127.31.131.126:34013)
0507 02:11:33.411254 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:33.411256 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:33.411264 (+ 8us) server_negotiation.cc:407] Connection header received
0507 02:11:33.411353 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:33.411354 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:33.411364 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:33.411417 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:33.411927 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:33.412352 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:33.412847 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:33.413012 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:33.414862 (+ 1850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:33.414866 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:33.414866 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:11:33.414873 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:33.417803 (+ 2930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:33.418073 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:33.418075 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:33.418075 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:33.418096 (+ 21us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:33.418217 (+ 121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:33.418217 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:33.418218 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:33.418298 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:33.418417 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:33.418474 (+ 57us) server_negotiation.cc:299] Negotiation successful
0507 02:11:33.418579 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20240507 02:11:33.419291 22686 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7615f7659f344c2aaafd8c723339b4e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f621d02b2c7d4f1799d6549aa5c9a6e0" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 34013 } } peers { permanent_uuid: "c4d97049ef4f470bab82a54a275f100b" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 38391 } } peers { permanent_uuid: "7615f7659f344c2aaafd8c723339b4e8" member_type: VOTER last_known_addr { host: "127.31.131.124" port: 44023 } }
I20240507 02:11:33.419418 22686 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7615f7659f344c2aaafd8c723339b4e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7615f7659f344c2aaafd8c723339b4e8, State: Initialized, Role: FOLLOWER
I20240507 02:11:33.419488 22686 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7615f7659f344c2aaafd8c723339b4e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f621d02b2c7d4f1799d6549aa5c9a6e0" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 34013 } } peers { permanent_uuid: "c4d97049ef4f470bab82a54a275f100b" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 38391 } } peers { permanent_uuid: "7615f7659f344c2aaafd8c723339b4e8" member_type: VOTER last_known_addr { host: "127.31.131.124" port: 44023 } }
I20240507 02:11:33.419728 22693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7615f7659f344c2aaafd8c723339b4e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f621d02b2c7d4f1799d6549aa5c9a6e0" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 34013 } } peers { permanent_uuid: "c4d97049ef4f470bab82a54a275f100b" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 38391 } } peers { permanent_uuid: "7615f7659f344c2aaafd8c723339b4e8" member_type: VOTER last_known_addr { host: "127.31.131.124" port: 44023 } } }
I20240507 02:11:33.419770 22686 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7615f7659f344c2aaafd8c723339b4e8 [sys.catalog]: configured and running, proceeding with master startup.
I20240507 02:11:33.419840 22693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7615f7659f344c2aaafd8c723339b4e8 [sys.catalog]: This master's current role is: FOLLOWER
W20240507 02:11:33.421607 22704 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7615f7659f344c2aaafd8c723339b4e8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240507 02:11:33.421851 22704 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
May 07 02:11:33 dist-test-slave-v0bg krb5kdc[22455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.131.125@KRBTEST.COM
I20240507 02:11:33.422247 22618 master_runner.cc:186] Error getting master registration for 127.31.131.126:34013: OK, instance_id { permanent_uuid: "f621d02b2c7d4f1799d6549aa5c9a6e0" instance_seqno: 1715047893113157 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20240507 02:11:33.423849 22618 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240507 02:11:33.426012 22615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:33.419293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41484 (local address 127.31.131.125:38391)
0507 02:11:33.419341 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:33.419343 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:33.419349 (+ 6us) server_negotiation.cc:407] Connection header received
0507 02:11:33.419372 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:33.419373 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:33.419382 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:33.419426 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:33.419941 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:33.420578 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:33.421304 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:33.421481 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:33.423626 (+ 2145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:33.423630 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:33.423631 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:11:33.423638 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:33.425172 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:33.425431 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:33.425439 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:33.425439 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:33.425464 (+ 25us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:33.425575 (+ 111us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:33.425575 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:33.425576 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:33.425684 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:33.425802 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:33.425836 (+ 34us) server_negotiation.cc:299] Negotiation successful
0507 02:11:33.425916 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20240507 02:11:33.427470 32269 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20240507 02:11:33.428233 22510 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "f621d02b2c7d4f1799d6549aa5c9a6e0"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46372
I20240507 02:11:33.452586 22536 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.131.124" port: 44023 } attempt: 2
May 07 02:11:33 dist-test-slave-v0bg krb5kdc[22455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047893, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.126@KRBTEST.COM for kudu/127.31.131.124@KRBTEST.COM
I20240507 02:11:33.459369 22691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:33.453046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53316 (local address 127.31.131.124:44023)
0507 02:11:33.453103 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:33.453105 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:33.453115 (+ 10us) server_negotiation.cc:407] Connection header received
0507 02:11:33.453143 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:33.453144 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:33.453155 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:33.453225 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:33.453719 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:33.454450 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:33.455199 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:33.455365 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:33.457011 (+ 1646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:33.457016 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:33.457016 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:11:33.457024 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:33.458367 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:33.458706 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:33.458707 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:33.458707 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:33.458731 (+ 24us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:33.458852 (+ 121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:33.458853 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:33.458853 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:33.459000 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:33.459057 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:33.459191 (+ 134us) server_negotiation.cc:299] Negotiation successful
0507 02:11:33.459272 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20240507 02:11:33.459369 22538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:33.452954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.124:44023 (local address 127.0.0.1:53316)
0507 02:11:33.453017 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:33.453027 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0507 02:11:33.453068 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:11:33.453238 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:11:33.453239 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:11:33.453435 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:11:33.453637 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:33.453640 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:33.454537 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:11:33.454538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:11:33.455116 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:33.455119 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:33.455191 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:33.455619 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:11:33.455624 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:11:33.456918 (+ 1294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:11:33.458437 (+ 1519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:11:33.458438 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:11:33.458438 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:11:33.458638 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:11:33.458773 (+ 135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:11:33.458774 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:11:33.458774 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:11:33.458801 (+ 27us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:11:33.459052 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:11:33.459053 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:11:33.459100 (+ 47us) client_negotiation.cc:769] Sending connection context
0507 02:11:33.459167 (+ 67us) client_negotiation.cc:240] Negotiation successful
0507 02:11:33.459260 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20240507 02:11:33.460824 22536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f621d02b2c7d4f1799d6549aa5c9a6e0: Bootstrap starting.
I20240507 02:11:33.461483 22536 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f621d02b2c7d4f1799d6549aa5c9a6e0: Neither blocks nor log segments found. Creating new log.
I20240507 02:11:33.461736 22536 log.cc:824] T 00000000000000000000000000000000 P f621d02b2c7d4f1799d6549aa5c9a6e0: Log is configured to *not* fsync() on all Append() calls
I20240507 02:11:33.462584 22536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f621d02b2c7d4f1799d6549aa5c9a6e0: No bootstrap required, opened a new log
I20240507 02:11:33.463719 22536 raft_consensus.cc:348] T 00000000000000000000000000000000 P f621d02b2c7d4f1799d6549aa5c9a6e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f621d02b2c7d4f1799d6549aa5c9a6e0" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 34013 } } peers { permanent_uuid: "c4d97049ef4f470bab82a54a275f100b" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 38391 } } peers { permanent_uuid: "7615f7659f344c2aaafd8c723339b4e8" member_type: VOTER last_known_addr { host: "127.31.131.124" port: 44023 } }
I20240507 02:11:33.463848 22536 raft_consensus.cc:729] T 00000000000000000000000000000000 P f621d02b2c7d4f1799d6549aa5c9a6e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f621d02b2c7d4f1799d6549aa5c9a6e0, State: Initialized, Role: FOLLOWER
I20240507 02:11:33.463917 22536 consensus_queue.cc:260] T 00000000000000000000000000000000 P f621d02b2c7d4f1799d6549aa5c9a6e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f621d02b2c7d4f1799d6549aa5c9a6e0" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 34013 } } peers { permanent_uuid: "c4d97049ef4f470bab82a54a275f100b" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 38391 } } peers { permanent_uuid: "7615f7659f344c2aaafd8c723339b4e8" member_type: VOTER last_known_addr { host: "127.31.131.124" port: 44023 } }
I20240507 02:11:33.464205 22536 sys_catalog.cc:564] T 00000000000000000000000000000000 P f621d02b2c7d4f1799d6549aa5c9a6e0 [sys.catalog]: configured and running, proceeding with master startup.
I20240507 02:11:33.464159 22707 sys_catalog.cc:455] T 00000000000000000000000000000000 P f621d02b2c7d4f1799d6549aa5c9a6e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f621d02b2c7d4f1799d6549aa5c9a6e0" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 34013 } } peers { permanent_uuid: "c4d97049ef4f470bab82a54a275f100b" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 38391 } } peers { permanent_uuid: "7615f7659f344c2aaafd8c723339b4e8" member_type: VOTER last_known_addr { host: "127.31.131.124" port: 44023 } } }
I20240507 02:11:33.464268 22707 sys_catalog.cc:458] T 00000000000000000000000000000000 P f621d02b2c7d4f1799d6549aa5c9a6e0 [sys.catalog]: This master's current role is: FOLLOWER
I20240507 02:11:33.464346 22510 raft_consensus.cc:482] T 00000000000000000000000000000000 P f621d02b2c7d4f1799d6549aa5c9a6e0 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20240507 02:11:33.464406 22510 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f621d02b2c7d4f1799d6549aa5c9a6e0 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:11:33.465083 22510 raft_consensus.cc:504] T 00000000000000000000000000000000 P f621d02b2c7d4f1799d6549aa5c9a6e0 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f621d02b2c7d4f1799d6549aa5c9a6e0" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 34013 } } peers { permanent_uuid: "c4d97049ef4f470bab82a54a275f100b" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 38391 } } peers { permanent_uuid: "7615f7659f344c2aaafd8c723339b4e8" member_type: VOTER last_known_addr { host: "127.31.131.124" port: 44023 } }
I20240507 02:11:33.465296 22510 leader_election.cc:288] T 00000000000000000000000000000000 P f621d02b2c7d4f1799d6549aa5c9a6e0 [CANDIDATE]: Term 1 election: Requested vote from peers c4d97049ef4f470bab82a54a275f100b (127.31.131.125:38391), 7615f7659f344c2aaafd8c723339b4e8 (127.31.131.124:44023)
I20240507 02:11:33.465596 22660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f621d02b2c7d4f1799d6549aa5c9a6e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7615f7659f344c2aaafd8c723339b4e8"
I20240507 02:11:33.465710 22660 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7615f7659f344c2aaafd8c723339b4e8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 02:11:33.466564 22660 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7615f7659f344c2aaafd8c723339b4e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f621d02b2c7d4f1799d6549aa5c9a6e0 in term 1.
I20240507 02:11:33.467033 22484 leader_election.cc:302] T 00000000000000000000000000000000 P f621d02b2c7d4f1799d6549aa5c9a6e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7615f7659f344c2aaafd8c723339b4e8, f621d02b2c7d4f1799d6549aa5c9a6e0; no voters:
I20240507 02:11:33.467226 22707 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f621d02b2c7d4f1799d6549aa5c9a6e0 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 02:11:33.467381 22707 raft_consensus.cc:686] T 00000000000000000000000000000000 P f621d02b2c7d4f1799d6549aa5c9a6e0 [term 1 LEADER]: Becoming Leader. State: Replica: f621d02b2c7d4f1799d6549aa5c9a6e0, State: Running, Role: LEADER
I20240507 02:11:33.467468 22707 consensus_queue.cc:237] T 00000000000000000000000000000000 P f621d02b2c7d4f1799d6549aa5c9a6e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f621d02b2c7d4f1799d6549aa5c9a6e0" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 34013 } } peers { permanent_uuid: "c4d97049ef4f470bab82a54a275f100b" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 38391 } } peers { permanent_uuid: "7615f7659f344c2aaafd8c723339b4e8" member_type: VOTER last_known_addr { host: "127.31.131.124" port: 44023 } }
I20240507 02:11:33.468207 22725 sys_catalog.cc:455] T 00000000000000000000000000000000 P f621d02b2c7d4f1799d6549aa5c9a6e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f621d02b2c7d4f1799d6549aa5c9a6e0. Latest consensus state: current_term: 1 leader_uuid: "f621d02b2c7d4f1799d6549aa5c9a6e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f621d02b2c7d4f1799d6549aa5c9a6e0" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 34013 } } peers { permanent_uuid: "c4d97049ef4f470bab82a54a275f100b" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 38391 } } peers { permanent_uuid: "7615f7659f344c2aaafd8c723339b4e8" member_type: VOTER last_known_addr { host: "127.31.131.124" port: 44023 } } }
I20240507 02:11:33.468469 22725 sys_catalog.cc:458] T 00000000000000000000000000000000 P f621d02b2c7d4f1799d6549aa5c9a6e0 [sys.catalog]: This master's current role is: LEADER
I20240507 02:11:33.469138 22468 master_runner.cc:186] Error getting master registration for 127.31.131.125:38391: OK, instance_id { permanent_uuid: "c4d97049ef4f470bab82a54a275f100b" instance_seqno: 1715047893252634 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20240507 02:11:33.471275 22468 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240507 02:11:33.471622 22727 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f621d02b2c7d4f1799d6549aa5c9a6e0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240507 02:11:33.471669 22727 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240507 02:11:33.471768 22728 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 02:11:33.471894 22728 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240507 02:11:33.476544 22615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:33.466389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41498 (local address 127.31.131.125:38391)
0507 02:11:33.466432 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:33.466434 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:33.466969 (+ 535us) server_negotiation.cc:407] Connection header received
0507 02:11:33.467087 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:33.467088 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:33.467113 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:33.467143 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:33.469591 (+ 2448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:33.470249 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:33.471207 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:33.471379 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:33.473527 (+ 2148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:33.473530 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:33.473531 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:11:33.473537 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:33.475089 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:33.475546 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:33.475547 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:33.475547 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:33.475587 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:33.475871 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:33.475872 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:33.475873 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:33.475996 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:33.476070 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:33.476363 (+ 293us) server_negotiation.cc:299] Negotiation successful
0507 02:11:33.476438 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20240507 02:11:33.478231 22660 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7615f7659f344c2aaafd8c723339b4e8 [term 1 FOLLOWER]: Refusing update from remote peer f621d02b2c7d4f1799d6549aa5c9a6e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 02:11:33.478632 22707 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f621d02b2c7d4f1799d6549aa5c9a6e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7615f7659f344c2aaafd8c723339b4e8" member_type: VOTER last_known_addr { host: "127.31.131.124" port: 44023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 02:11:33.480026 22693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7615f7659f344c2aaafd8c723339b4e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f621d02b2c7d4f1799d6549aa5c9a6e0. Latest consensus state: current_term: 1 leader_uuid: "f621d02b2c7d4f1799d6549aa5c9a6e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f621d02b2c7d4f1799d6549aa5c9a6e0" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 34013 } } peers { permanent_uuid: "c4d97049ef4f470bab82a54a275f100b" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 38391 } } peers { permanent_uuid: "7615f7659f344c2aaafd8c723339b4e8" member_type: VOTER last_known_addr { host: "127.31.131.124" port: 44023 } } }
I20240507 02:11:33.480110 22693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7615f7659f344c2aaafd8c723339b4e8 [sys.catalog]: This master's current role is: FOLLOWER
I20240507 02:11:33.480412 22539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:33.466358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46378 (local address 127.31.131.126:34013)
0507 02:11:33.466394 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:33.466396 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:33.466897 (+ 501us) server_negotiation.cc:407] Connection header received
0507 02:11:33.467107 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:33.467108 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:33.467117 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:33.467143 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:33.469692 (+ 2549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:33.470324 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:33.471519 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:33.471679 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:33.475038 (+ 3359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:33.475042 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:33.475042 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:11:33.475049 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:33.476605 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:33.477689 (+ 1084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:33.477690 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:33.477690 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:33.477715 (+ 25us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:33.478965 (+ 1250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:33.478966 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:33.478966 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:33.479083 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:33.479148 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:33.480248 (+ 1100us) server_negotiation.cc:299] Negotiation successful
0507 02:11:33.480329 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20240507 02:11:33.481333 22707 sys_catalog.cc:455] T 00000000000000000000000000000000 P f621d02b2c7d4f1799d6549aa5c9a6e0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f621d02b2c7d4f1799d6549aa5c9a6e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f621d02b2c7d4f1799d6549aa5c9a6e0" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 34013 } } peers { permanent_uuid: "c4d97049ef4f470bab82a54a275f100b" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 38391 } } peers { permanent_uuid: "7615f7659f344c2aaafd8c723339b4e8" member_type: VOTER last_known_addr { host: "127.31.131.124" port: 44023 } } }
I20240507 02:11:33.481408 22707 sys_catalog.cc:458] T 00000000000000000000000000000000 P f621d02b2c7d4f1799d6549aa5c9a6e0 [sys.catalog]: This master's current role is: LEADER
I20240507 02:11:33.481542 22691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:33.466670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53324 (local address 127.31.131.124:44023)
0507 02:11:33.466703 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:33.466705 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:33.467066 (+ 361us) server_negotiation.cc:407] Connection header received
0507 02:11:33.468082 (+ 1016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:33.468083 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:33.468093 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:33.468121 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:33.468979 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:33.469614 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:33.471751 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:33.472243 (+ 492us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:33.476919 (+ 4676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:33.476924 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:33.476924 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:11:33.476930 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:33.478388 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:33.479376 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:33.479377 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:33.479377 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:33.479399 (+ 22us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:33.479777 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:33.479778 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:33.479789 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:33.479898 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:33.479989 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:33.481381 (+ 1392us) server_negotiation.cc:299] Negotiation successful
0507 02:11:33.481469 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20240507 02:11:33.481989 22693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7615f7659f344c2aaafd8c723339b4e8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f621d02b2c7d4f1799d6549aa5c9a6e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f621d02b2c7d4f1799d6549aa5c9a6e0" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 34013 } } peers { permanent_uuid: "c4d97049ef4f470bab82a54a275f100b" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 38391 } } peers { permanent_uuid: "7615f7659f344c2aaafd8c723339b4e8" member_type: VOTER last_known_addr { host: "127.31.131.124" port: 44023 } } }
I20240507 02:11:33.482070 22693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7615f7659f344c2aaafd8c723339b4e8 [sys.catalog]: This master's current role is: FOLLOWER
I20240507 02:11:33.482210 22730 mvcc.cc:205] Tried to move back new op lower bound from 7024836171684171776 to 7024836171643482112. Current Snapshot: MvccSnapshot[applied={T|T < 7024836171684171776}]
I20240507 02:11:33.482954 22728 catalog_manager.cc:1347] Generated new cluster ID: 84e2de82f8554823b98b851e6cb11b85
I20240507 02:11:33.483003 22728 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240507 02:11:33.482970 22731 mvcc.cc:205] Tried to move back new op lower bound from 7024836171684171776 to 7024836171643482112. Current Snapshot: MvccSnapshot[applied={T|T < 7024836171684171776}]
I20240507 02:11:33.486583 22611 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.131.124" port: 44023 } attempt: 3
May 07 02:11:33 dist-test-slave-v0bg krb5kdc[22455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1715047893, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.131.125@KRBTEST.COM for kudu/127.31.131.124@KRBTEST.COM
I20240507 02:11:33.493883 22613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:33.486960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.131.124:44023 (local address 127.0.0.1:53340)
0507 02:11:33.487156 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0507 02:11:33.487168 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0507 02:11:33.487207 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0507 02:11:33.487425 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0507 02:11:33.487426 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0507 02:11:33.487599 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0507 02:11:33.487786 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:33.487788 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:33.488619 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0507 02:11:33.488620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0507 02:11:33.489134 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0507 02:11:33.489135 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0507 02:11:33.489187 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:33.489912 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0507 02:11:33.489916 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0507 02:11:33.491278 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0507 02:11:33.493108 (+ 1830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:11:33.493109 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:11:33.493109 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:11:33.493270 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:11:33.493389 (+ 119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0507 02:11:33.493390 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0507 02:11:33.493391 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0507 02:11:33.493416 (+ 25us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0507 02:11:33.493611 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0507 02:11:33.493612 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0507 02:11:33.493651 (+ 39us) client_negotiation.cc:769] Sending connection context
0507 02:11:33.493705 (+ 54us) client_negotiation.cc:240] Negotiation successful
0507 02:11:33.493788 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180}
I20240507 02:11:33.494225 22691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:33.486969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53340 (local address 127.31.131.124:44023)
0507 02:11:33.487001 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:33.487003 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:33.487338 (+ 335us) server_negotiation.cc:407] Connection header received
0507 02:11:33.487355 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:33.487356 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:33.487365 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:33.487420 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:33.487896 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:33.488540 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:33.489512 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:33.489682 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:33.491361 (+ 1679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0507 02:11:33.491365 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0507 02:11:33.491365 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0507 02:11:33.491370 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0507 02:11:33.493055 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:33.493329 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:33.493330 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:33.493330 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:33.493350 (+ 20us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0507 02:11:33.493471 (+ 121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0507 02:11:33.493472 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0507 02:11:33.493472 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0507 02:11:33.493562 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0507 02:11:33.493610 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:33.494047 (+ 437us) server_negotiation.cc:299] Negotiation successful
0507 02:11:33.494138 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20240507 02:11:33.495499 22611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4d97049ef4f470bab82a54a275f100b: Bootstrap starting.
I20240507 02:11:33.496069 22611 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4d97049ef4f470bab82a54a275f100b: Neither blocks nor log segments found. Creating new log.
I20240507 02:11:33.496326 22611 log.cc:824] T 00000000000000000000000000000000 P c4d97049ef4f470bab82a54a275f100b: Log is configured to *not* fsync() on all Append() calls
I20240507 02:11:33.497339 22611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4d97049ef4f470bab82a54a275f100b: No bootstrap required, opened a new log
I20240507 02:11:33.499008 22611 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4d97049ef4f470bab82a54a275f100b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f621d02b2c7d4f1799d6549aa5c9a6e0" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 34013 } } peers { permanent_uuid: "c4d97049ef4f470bab82a54a275f100b" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 38391 } } peers { permanent_uuid: "7615f7659f344c2aaafd8c723339b4e8" member_type: VOTER last_known_addr { host: "127.31.131.124" port: 44023 } }
I20240507 02:11:33.499111 22611 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4d97049ef4f470bab82a54a275f100b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4d97049ef4f470bab82a54a275f100b, State: Initialized, Role: FOLLOWER
I20240507 02:11:33.499161 22611 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4d97049ef4f470bab82a54a275f100b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f621d02b2c7d4f1799d6549aa5c9a6e0" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 34013 } } peers { permanent_uuid: "c4d97049ef4f470bab82a54a275f100b" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 38391 } } peers { permanent_uuid: "7615f7659f344c2aaafd8c723339b4e8" member_type: VOTER last_known_addr { host: "127.31.131.124" port: 44023 } }
I20240507 02:11:33.499383 22611 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4d97049ef4f470bab82a54a275f100b [sys.catalog]: configured and running, proceeding with master startup.
I20240507 02:11:33.500564 22584 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4d97049ef4f470bab82a54a275f100b [term 0 FOLLOWER]: Advancing to term 1
W20240507 02:11:33.500890 22564 tablet.cc:2369] T 00000000000000000000000000000000 P c4d97049ef4f470bab82a54a275f100b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240507 02:11:33.501685 22584 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c4d97049ef4f470bab82a54a275f100b [term 1 FOLLOWER]: Refusing update from remote peer f621d02b2c7d4f1799d6549aa5c9a6e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 02:11:33.502228 22585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f621d02b2c7d4f1799d6549aa5c9a6e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c4d97049ef4f470bab82a54a275f100b"
I20240507 02:11:33.502907 22585 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c4d97049ef4f470bab82a54a275f100b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f621d02b2c7d4f1799d6549aa5c9a6e0 in term 1.
I20240507 02:11:33.503139 22725 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f621d02b2c7d4f1799d6549aa5c9a6e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4d97049ef4f470bab82a54a275f100b" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240507 02:11:33.506724 22747 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c4d97049ef4f470bab82a54a275f100b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240507 02:11:33.506806 22747 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240507 02:11:33.507124 22749 mvcc.cc:205] Tried to move back new op lower bound from 7024836171684171776 to 7024836171643482112. Current Snapshot: MvccSnapshot[applied={T|T < 7024836171684171776}]
I20240507 02:11:33.507364 22748 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4d97049ef4f470bab82a54a275f100b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f621d02b2c7d4f1799d6549aa5c9a6e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f621d02b2c7d4f1799d6549aa5c9a6e0" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 34013 } } peers { permanent_uuid: "c4d97049ef4f470bab82a54a275f100b" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 38391 } } peers { permanent_uuid: "7615f7659f344c2aaafd8c723339b4e8" member_type: VOTER last_known_addr { host: "127.31.131.124" port: 44023 } } }
I20240507 02:11:33.507498 22748 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4d97049ef4f470bab82a54a275f100b [sys.catalog]: This master's current role is: FOLLOWER
I20240507 02:11:33.507566 22748 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4d97049ef4f470bab82a54a275f100b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f621d02b2c7d4f1799d6549aa5c9a6e0. Latest consensus state: current_term: 1 leader_uuid: "f621d02b2c7d4f1799d6549aa5c9a6e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f621d02b2c7d4f1799d6549aa5c9a6e0" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 34013 } } peers { permanent_uuid: "c4d97049ef4f470bab82a54a275f100b" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 38391 } } peers { permanent_uuid: "7615f7659f344c2aaafd8c723339b4e8" member_type: VOTER last_known_addr { host: "127.31.131.124" port: 44023 } } }
I20240507 02:11:33.507640 22748 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4d97049ef4f470bab82a54a275f100b [sys.catalog]: This master's current role is: FOLLOWER
I20240507 02:11:33.507694 22748 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4d97049ef4f470bab82a54a275f100b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f621d02b2c7d4f1799d6549aa5c9a6e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f621d02b2c7d4f1799d6549aa5c9a6e0" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 34013 } } peers { permanent_uuid: "c4d97049ef4f470bab82a54a275f100b" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 38391 } } peers { permanent_uuid: "7615f7659f344c2aaafd8c723339b4e8" member_type: VOTER last_known_addr { host: "127.31.131.124" port: 44023 } } }
I20240507 02:11:33.507786 22748 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4d97049ef4f470bab82a54a275f100b [sys.catalog]: This master's current role is: FOLLOWER
I20240507 02:11:33.525254 22725 sys_catalog.cc:455] T 00000000000000000000000000000000 P f621d02b2c7d4f1799d6549aa5c9a6e0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f621d02b2c7d4f1799d6549aa5c9a6e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f621d02b2c7d4f1799d6549aa5c9a6e0" member_type: VOTER last_known_addr { host: "127.31.131.126" port: 34013 } } peers { permanent_uuid: "c4d97049ef4f470bab82a54a275f100b" member_type: VOTER last_known_addr { host: "127.31.131.125" port: 38391 } } peers { permanent_uuid: "7615f7659f344c2aaafd8c723339b4e8" member_type: VOTER last_known_addr { host: "127.31.131.124" port: 44023 } } }
I20240507 02:11:33.525372 22725 sys_catalog.cc:458] T 00000000000000000000000000000000 P f621d02b2c7d4f1799d6549aa5c9a6e0 [sys.catalog]: This master's current role is: LEADER
I20240507 02:11:33.530635 22728 catalog_manager.cc:1370] Generated new certificate authority record
I20240507 02:11:33.536242 22728 catalog_manager.cc:1504] Loading token signing keys...
I20240507 02:11:33.546988 22728 catalog_manager.cc:5857] T 00000000000000000000000000000000 P f621d02b2c7d4f1799d6549aa5c9a6e0: Generated new TSK 0
I20240507 02:11:33.547235 22728 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20240507 02:11:33.571650 22615 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:11:33.569976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41506 (local address 127.31.131.125:38391)
0507 02:11:33.570016 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:33.570018 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:33.570789 (+ 771us) server_negotiation.cc:407] Connection header received
0507 02:11:33.571012 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:33.571013 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:33.571026 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:33.571059 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:33.571575 (+ 516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41506: BlockingRecv error: recv got EOF from 127.0.0.1:41506 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20240507 02:11:33.572290 22539 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:11:33.569702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46394 (local address 127.31.131.126:34013)
0507 02:11:33.569753 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:33.569755 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:33.570896 (+ 1141us) server_negotiation.cc:407] Connection header received
0507 02:11:33.571820 (+ 924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:33.571821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:33.571834 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:33.571874 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:33.572231 (+ 357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46394: BlockingRecv error: recv got EOF from 127.0.0.1:46394 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20240507 02:11:33.573110 22691 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:11:33.570473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53352 (local address 127.31.131.124:44023)
0507 02:11:33.570518 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:33.570520 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:33.572572 (+ 2052us) server_negotiation.cc:407] Connection header received
0507 02:11:33.572688 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:33.572689 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:33.572701 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:33.572733 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:33.573056 (+ 323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53352: BlockingRecv error: recv got EOF from 127.0.0.1:53352 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20240507 02:11:33.575575 22615 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:11:33.574721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41518 (local address 127.31.131.125:38391)
0507 02:11:33.574756 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:33.574757 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:33.575007 (+ 250us) server_negotiation.cc:407] Connection header received
0507 02:11:33.575125 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:33.575125 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:33.575142 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:33.575174 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:33.575527 (+ 353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41518: BlockingRecv error: recv got EOF from 127.0.0.1:41518 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20240507 02:11:33.579644 22615 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:11:33.578906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41528 (local address 127.31.131.125:38391)
0507 02:11:33.578939 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:33.578940 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:33.579166 (+ 226us) server_negotiation.cc:407] Connection header received
0507 02:11:33.579264 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:33.579265 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:33.579276 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:33.579306 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:33.579598 (+ 292us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41528: BlockingRecv error: recv got EOF from 127.0.0.1:41528 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20240507 02:11:33.583525 22615 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:11:33.582767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41542 (local address 127.31.131.125:38391)
0507 02:11:33.582799 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:33.582800 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:33.583058 (+ 258us) server_negotiation.cc:407] Connection header received
0507 02:11:33.583155 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:33.583156 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:33.583166 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:33.583196 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:33.583484 (+ 288us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41542: BlockingRecv error: recv got EOF from 127.0.0.1:41542 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20240507 02:11:33.589484 22615 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:11:33.588711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41558 (local address 127.31.131.125:38391)
0507 02:11:33.588745 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:33.588747 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:33.589000 (+ 253us) server_negotiation.cc:407] Connection header received
0507 02:11:33.589098 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:33.589099 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:33.589110 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:33.589140 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:33.589440 (+ 300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41558: BlockingRecv error: recv got EOF from 127.0.0.1:41558 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20240507 02:11:33.599829 22615 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:11:33.598788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41574 (local address 127.31.131.125:38391)
0507 02:11:33.598825 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:33.598827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:33.599159 (+ 332us) server_negotiation.cc:407] Connection header received
0507 02:11:33.599265 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:33.599266 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:33.599277 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:33.599308 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:33.599774 (+ 466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41574: BlockingRecv error: recv got EOF from 127.0.0.1:41574 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20240507 02:11:33.618023 22615 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:11:33.617188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41588 (local address 127.31.131.125:38391)
0507 02:11:33.617230 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:33.617232 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:33.617491 (+ 259us) server_negotiation.cc:407] Connection header received
0507 02:11:33.617582 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:33.617583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:33.617594 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:33.617620 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:33.617963 (+ 343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41588: BlockingRecv error: recv got EOF from 127.0.0.1:41588 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20240507 02:11:33.652144 22615 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:11:33.651393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41602 (local address 127.31.131.125:38391)
0507 02:11:33.651445 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:33.651447 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:33.651586 (+ 139us) server_negotiation.cc:407] Connection header received
0507 02:11:33.651647 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:33.651648 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:33.651659 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:33.651733 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:33.652086 (+ 353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41602: BlockingRecv error: recv got EOF from 127.0.0.1:41602 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20240507 02:11:33.718030 22615 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:11:33.717400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41604 (local address 127.31.131.125:38391)
0507 02:11:33.717460 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:33.717462 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:33.717577 (+ 115us) server_negotiation.cc:407] Connection header received
0507 02:11:33.717631 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:33.717632 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:33.717642 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:33.717692 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:33.717969 (+ 277us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41604: BlockingRecv error: recv got EOF from 127.0.0.1:41604 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20240507 02:11:33.847798 22615 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:11:33.847147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41608 (local address 127.31.131.125:38391)
0507 02:11:33.847203 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:33.847206 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:33.847294 (+ 88us) server_negotiation.cc:407] Connection header received
0507 02:11:33.847347 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:33.847348 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:33.847358 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:33.847426 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:33.847735 (+ 309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41608: BlockingRecv error: recv got EOF from 127.0.0.1:41608 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20240507 02:11:34.105435 22615 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 02:11:34.104909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41618 (local address 127.31.131.125:38391)
0507 02:11:34.104969 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:34.104971 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:34.104983 (+ 12us) server_negotiation.cc:407] Connection header received
0507 02:11:34.105045 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:34.105046 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:34.105056 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:34.105103 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0507 02:11:34.105374 (+ 271us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41618: BlockingRecv error: recv got EOF from 127.0.0.1:41618 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20240507 02:11:34.422358 22704 catalog_manager.cc:1259] Loaded cluster ID: 84e2de82f8554823b98b851e6cb11b85
I20240507 02:11:34.422446 22704 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7615f7659f344c2aaafd8c723339b4e8: loading cluster ID for follower catalog manager: success
I20240507 02:11:34.423303 22704 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7615f7659f344c2aaafd8c723339b4e8: acquiring CA information for follower catalog manager: success
I20240507 02:11:34.423591 22704 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7615f7659f344c2aaafd8c723339b4e8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240507 02:11:34.507198 22747 catalog_manager.cc:1259] Loaded cluster ID: 84e2de82f8554823b98b851e6cb11b85
I20240507 02:11:34.507266 22747 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c4d97049ef4f470bab82a54a275f100b: loading cluster ID for follower catalog manager: success
I20240507 02:11:34.508134 22747 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c4d97049ef4f470bab82a54a275f100b: acquiring CA information for follower catalog manager: success
I20240507 02:11:34.508419 22747 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c4d97049ef4f470bab82a54a275f100b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240507 02:11:34.620986 22818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:34.618630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41630 (local address 127.31.131.125:38391)
0507 02:11:34.618748 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:34.618750 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:34.618788 (+ 38us) server_negotiation.cc:407] Connection header received
0507 02:11:34.618881 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:34.618881 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:34.618936 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:34.619008 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0507 02:11:34.619284 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:34.619798 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:34.620458 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:34.620599 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:34.620641 (+ 42us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0507 02:11:34.620731 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0507 02:11:34.620808 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:34.620875 (+ 67us) server_negotiation.cc:299] Negotiation successful
0507 02:11:34.620904 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":58,"threads_started":1}
I20240507 02:11:34.624374 22824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:34.622474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53354 (local address 127.31.131.124:44023)
0507 02:11:34.622614 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:34.622616 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:34.622668 (+ 52us) server_negotiation.cc:407] Connection header received
0507 02:11:34.622747 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:34.622748 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:34.622772 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:34.622842 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0507 02:11:34.623035 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:34.623512 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:34.623988 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:34.624095 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:34.624125 (+ 30us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0507 02:11:34.624202 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0507 02:11:34.624257 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:34.624301 (+ 44us) server_negotiation.cc:299] Negotiation successful
0507 02:11:34.624320 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":63,"threads_started":1}
I20240507 02:11:34.628271 22831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:34.625864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46400 (local address 127.31.131.126:34013)
0507 02:11:34.626109 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:34.626111 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:34.626119 (+ 8us) server_negotiation.cc:407] Connection header received
0507 02:11:34.626167 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:34.626168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:34.626200 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:34.626286 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0507 02:11:34.626768 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:34.627250 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:34.627782 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:34.627910 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:34.627956 (+ 46us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0507 02:11:34.628020 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0507 02:11:34.628074 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:34.628161 (+ 87us) server_negotiation.cc:299] Negotiation successful
0507 02:11:34.628193 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":55,"threads_started":1}
I20240507 02:11:34.628646 22818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:34.625816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41642 (local address 127.31.131.125:38391)
0507 02:11:34.625962 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:34.625964 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:34.626068 (+ 104us) server_negotiation.cc:407] Connection header received
0507 02:11:34.626153 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:34.626154 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:34.626165 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:34.626210 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0507 02:11:34.626518 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:34.627267 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:34.628057 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:34.628218 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:34.628239 (+ 21us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0507 02:11:34.628258 (+ 19us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0507 02:11:34.628302 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:34.628551 (+ 249us) server_negotiation.cc:299] Negotiation successful
0507 02:11:34.628580 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20240507 02:11:34.628901 22824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0507 02:11:34.626039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53364 (local address 127.31.131.124:44023)
0507 02:11:34.626085 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0507 02:11:34.626087 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0507 02:11:34.626094 (+ 7us) server_negotiation.cc:407] Connection header received
0507 02:11:34.626206 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0507 02:11:34.626207 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0507 02:11:34.626220 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0507 02:11:34.626279 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0507 02:11:34.626529 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:34.627262 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0507 02:11:34.628331 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0507 02:11:34.628489 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0507 02:11:34.628508 (+ 19us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0507 02:11:34.628532 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0507 02:11:34.628571 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0507 02:11:34.628828 (+ 257us) server_negotiation.cc:299] Negotiation successful
0507 02:11:34.628851 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20240507 02:11:34.630007 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 22468
I20240507 02:11:34.634821 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 22543
I20240507 02:11:34.638826 32269 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTztKno/build/release/bin/kudu with pid 22618
2024-05-07T02:11:34Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2619 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5242 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (290208 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
1 FAILED TEST